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FOREWORD 

Evaluation has come a long way in education during the past 
five years. Suddenly, it seems, everyone responsible for programs 
has become conscious of evaluation within the context of accountability, 
as .well as improvement of their programs. In addition, research and 
development projects sponsored by the Federal Government, and other 
agencies, have had provisions for evaluations included in the contracts. 

At the same time, increasing numbers of scholars h^ave been re- 
examining the existing evaluation models, have been modifying them and 
proposing new versions. To the credit of the field, adult educators 
have been among those fn the forefront adding to the development of 
evaluation models. 

Perhaps the most pressing need for evaluation models in adult 
education has been in the area of programs for disadvantaged adults. 
Recognizing this need, the ERIC Clearinghouse on Adult Education took 
the initiative to bring together the many contemporary approaches to 
program evaluation as they can Be applied to programs for disadvantaged 
adults. To this end, the Clearinghouse commissioned Dr. Sara M. Steele 
to produce a monograph on this topic. 

After a year's concentrated^ work. Dr. Steele and her team of 
colleagues at the University of Wisconsin, produced this monograph. 
Nearly fifty scholars and practitioners provided critical comments on 
the various drafts of this monograph. The end product is a rather 
comprehensive compilation of evaluation models suitable for programs 
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for disadvantaged adults, plus some practical guidelines for 
applying the models to various situation which may confront a 
practitioner. 

ERIC/AE takes pride in presenting this document, and is 
grateful to Sara Steele for her contribution tc the field. 



May 19, 1973 



Stanely M. Grabowski 
Director- 

ERIC Clearinghouse on Adult Educat 



PREFACE 



7his monograph is written for those responsible for designing evaluation 
strategy for an institution or agency • In some cases, this is the person who 
conducts the program. In most instances, however, it's an administrator or 
his assistants. Most of the intended audience has some knowledge of much 
of the material included. This publication, then, is a source of review and 
a reference for them. It may also be of use to other program personnel. 

Although it was contracted for in terms of evaluating recent social 
programs, much of the content originated in other fields and most of it is 
applicable regardless of the type of program that is being evaluated. 

The monograph is a major team effort. The help of the following people 

is acknowledged and greatly appreciated: 

O the Wisconsin production team: Merrill Ewert, project assistant; 
Colleen Schuh, editor; Alice Durnford, secretary; Linda Weeden, 
final typist; Barbara Schutz, Charlene Duraira j , and Ruth 
Jacobson, typists; Beverly Cunningham, administrative secretary. 

• Dr. Patrick Boyle and other administrators and colleagues here at 
the University of Wisconsin who arranged changes in responsibilities 
so that the contract could be undertaken and who gave support and 
encouragement throughout the project. 

O the authors who created and refined the concepts and ideas summarized 
in the monograph and the publishers who brought those ideas into 
circulation. 

O program personnel from throughout tha United States and Canada who 
fitted reviewing the first draft into their extremely busy schedules. 

O professors of adult and extension education who gave impetus to the 
development of the classification structure. 

# the Office of Education and ERIC Clearinghouse on Adult Education which 
contracted with the University of Wisconsin for the manuscript and, 

in particular. Dr. Stanley Grabowski, director of ERIC, who provided 
the opportunity for proposing the idea for the manuscript and then 
guided its development. 
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INTRODUCTION 

In the last 10 years many programs have emerged, such as adult basic educa- 
lion, manpower training, applied nutrition, human resource development, citizen 
education and leadership development, rehabilitation, that need to handle evalua- 
tion effectively and efficiently. The last five years have produced a host of 
ideas about what program evaluation is, what it should do,«and how it should 
be done. Ideas about evaluation are in flux and transition. Yet, the average 
person responsible for establishing concepts and strategies of evaluation at 
the program operations level seldom has time to track down and study new 
developments in evaluation. Nor,"^does he have time to analyze h'^w well the 
proposed concepts meet his own needs, to identify gaps, and to be in a position 
to help build more realistic approaches to evaluation. 

This monograph is designed as a reference to contemporary evaluation approaches, 
It brings together a variety of old^and new frameworks and ideas about evalua- 
tion and shows how they're related to one another. It identifies some of the 
literature on evaluation from a variety of fields and some of the people who 
are generating ideas about evaluation. It can serve you by: 

1. Bringing you up to date on what has been happening to evaluation concepts. 

2. Increasing your understanding of the present state of evaluative approaches, 

3. Providing a basis for supporting or improving the evaluative approaches 
you're now using. 

^. Bringing to your attention frameworks that can help you design evalua- 
tive activities. 

5. Providing background when you need to discuss evaluation with con- 
sultants . 

6. Identifying the areas where you need to be active and encourage others 
to contribute to further development of evaluation strategy and tech- 
niques that will be relevant and useful in evaluating your type of 
program. 
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The first section of the monograph pi'ovides an overview of the current 
situation* The last section presents brief summaries of contemporary models 
and approaches to evaluation. The middle section provides some guidelines 
for developing evaluation and a problem-focused index to the various 
models in the later section. 

Those who want to get an overview of what's happening in evaluation 
will find the first section most useful. Those who want to impro^'e or develop 
an evaluation strategy may want to skip the first section and go directly 
to Section II • However, some of the approaches included in the last section 
v;ill seem out of order unlei-s you have kept up on recent trends in evaluation 
literature or scanned Section I. 



Section I 

LOOKING AT THE CONTEMPORARY EVALUATION SCENE 



This section presents an overview of what has been happening. It 
(1) identifies some of the influences that particularly affect <t ..^ 

programs for disadvantaged adults, (2) summarizes major developments in 
the conceptualization of evaluation that have implications for those 
programs, (3) introduces some of the types of approach that evaluation is 
currently taking, and (4) suggests the kind of work that still needs to be 
done before evaluation will attain its full effectiveness as a programming 
tool. 

Needs and Issues in Evaluating 
Programs for Disadvantaged Adult s 

Heroic attempts have been made to evaluate programs for disadvantaged 
adults — those adults who are at a disadvantage,- whether socially, culturally, 
economically, or physically in terms of the majority. Yet, there's often a 
general feeling of uneasiness and frustration associated with such evaluation 
because it has often proved less than satisfactory. There's a growing 
realization that somethi..g's needed that traditional evaluation approaches 
haven't been able to give. 

Evaluation strategies are needed that woi?k well within the environment 
surrounding programs for the disadvantaged. Although -most of the needs and 
issues involved in evaluating such progi-ams may also be inherent in other 
adult education programs, they're much more visible and create more pressure 
and frustration in new or controversial programs~especially programs with 
a broad base of involvement of governmental agencies and the public. 



Examining some of the historical and environmental factors associated 
with some prograTT^** for the disadvantaged may help to clarify the kinds of 
evaluative acti ''y ,eded ana bring the causes of uneasiness and frustra- 
tion into sharper focus. 

Some of the influences that strongly affect evaluation of programs for 
the disadvantaged include: 

1. Variance in viewpoints and needs # 

2. Multiple units of programs. 

3. Influence of state and federal funding. 

4. Inherentness of conflict. 

5. Lack of past operating history. 

6. Relationship to established programs. 

7. Assumptions about successful programs. 

]' ^ ^iance in Viewpoints and Needs 

One major source of frustration is the lack of understanding of the 
number of types of people who have needs for evaluative information about 
programs and the extent to which these needs differ. Although all adult 
education programs have a potential for five types of people — local 
administrators J civic leaders and other influentials j administrators of 
funding agencies y program personnel^ and 'program participants —^o be 
interested in evaluation, in most situations one or more of these groups 
are relatively passive. The interplay is between the needs of only one or 
two of the groups. However, in programs for disadvantaged adults, not only 
are they all apt to be active, but the needs of a sixth group, social 
policy makers^ also exert pressure. 

Social Policy Makers* Because programs for the disadvantaged are 
enmeshed in social policy and are instruments for carrying out certain 



ERLC 



policies, evaluation is often requested and/or used by those trying to set 
or influence social policy—legislators who introduce and promote 
certain legislation, experts, politicians, leaders of causes. Those 
concerned with social policy want conclusive research data to defend or 
support policy decisions to show the value and effects of a program in 
relation to a particular social problem or policy. 

Civic Leaders and Other Inflnentials. Some influent ials have no direct 
connection with the funding of programs. Through their status position they 
do, however, influence prospective clientele and the kinds of support that 
programs receive. They're hard to identify. They may be bartenders, priests, 
street gang leaders, or cops or. the beat. They evaluate programs and affect 
the program by their support, opposition, or indifference. Less is known about 
the criteria they apply or the kind of information they use. But because 
their position of influence usually rests in part on their being on top of 
things and giving good advise to their supporters, they have needs related 
to program evaluation. 

Other influentials— ^those who serve in Congress, state legislators, 
or local governing bodies responsible for allocating public funds — want 
evaluation data as an input into funding decisions. Some are concerned only 
with the value their constituents attach to a program. Others scrutin5,2e 
the extent to which the program produces the results it*s supposed to. 

Ad/ninietratora of Funding Agencies. Administrators of the government 
or private organizations that allocate funds to programs not only need 
evaluation that helps them administer and manage their work, but also helps 
them provide the kind of evaluative information that the policy makers and 
other legislators Cor the board of directors in the case of foundations) 
want. They're the men in the middle. 



Local Adminietrators . Administration is valueless unless it's able to 
guide the unit to optimum fulfillment of mission. Local administrators need 
evaluative information not only to meet the requests of the funding agencies 
and as a basis for making internal decisions on budget, promotion, and 
salary increases, but also as a means of guiding staff in making basic 
program decisions. The administrator must take a direct and active role 
in program decisions. The administrator must take a direct and active role 
in program evaluation. He must use program evaluation as a base for providing 
administrative leadership, he also must be able to judge what evaluative 
strategies should have highest priority in helping his staff develop their 
own competency, in meeting the informational needs of outsiders, and in main- 
taining rapport with clientele. 

Progrcpn Personnel. Although the programmer has rf^al -concerns about 
examining the results of his program, he must also use the evaluative processes 
that will give him the kind of immediate feedback he can use to carry out a 
program. Continuous evaluation during the programming process is as or more 
important than retrospective analysis. 

While in many other adult education programs, programming is primarily 
a "one-man show" or a team operation of two or three professionals, programs 
for disadvantaged adults use a wide range of program personnel. Para- 
professionals are important. With their advent, the front-line professional 
has become partly an administrator and partly a direct programmer. In that 
role, he must not only be able to evaluate his own activities, but to help 
the paraprofessionals evaluate their work and draw them into a cooperative 
evaluation of the work of the entire tea^n. 

Too often program personnel are viewed as being one of the subjects of 
the evaluation or the provider of infomnation. Often, too little attention 
is given to the kinds of program evaluation that will be of real use to them 
in programming. 



Program Participants* The program participant receives even less 
recognition in evaluation schemes. His needs related to program evaluation 
are often bypassed. And yet, in programs for the disadvantaged, the evalua- 
tion made by the individual participant is of utmost importance. It determines 
his own activity. What he does — how he uses his energy, time, and limited 
money — are extremely important to him. He doesn't want to miss something 
that will be of value. On the other hand, he's unwilling to waste time on 
things that are neither relevant nor pleasant. 

Sometimes the kind of evaluative information needed by the six groups 
coincide. Often it differs. For example, the kind of information the po- 
tential participant uses in deciding whether to take part in a program is 
different from the research type data that the social policy maker is 
striving for. 

Not only does each of the six groups need evaluation information, .but 
in turn, each may also need to know what evaluations the other groups have 
made of the program. For example, the programmer, program administrator, 
administrator of the funding agency, and social policy maker need to know 
what judgments the clientele have formed about the program* If it's important 
to assess participant reaction in traditional programs where the participants 
are usually motivated toward participation, and the programmer is experienced 
in identifying signs and symptoms of reactions, then it's doubly important 
that we find uays to assess participant reactions when the participants may 
not be as highly motivated and when the programmer may not be well versed 
in reading natural signs. Hot only must a constant check be kept on How 
participants are reacting, but evidence of negative reaction must be fed 
back into the program immediately so adjustments can be made* The program 
unit is often more conscious of assessing the reactions of those to whom 
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they report, then in following participant reactions. This may be true even 
though negative reactions on the part of participants may result in no program 
participants and ultimately in no programs. However, which reactions to 
heed usually isn't an either-or situation. Mechanisms must be found for 
understanding the evaluative conclusions formed by everyone involved. There 
may be advantages in getting the interested parties in discussion that leads 
to the formation of mutual judgments. 

Local program personnel — administrators and programmers — find themselves 
in the complex role of: (1) supplying information that others will use in 
their evaluations, (2) trying to get and respond to information about the 
kinds of judgments others are making, and (3) trying to get evaluative 
information on program process and results that helps them make operating 
decisions. What they do in relation to any of these three roles must be done 
in conjunction with or as time away from their other program responsibilities. 
They must be able to balance and reconcile evaluative activities within the 
parameters of the small amount of resources they can allocate to evaluation. 

One way of reducing uneasiness, if not frustration s is clearly under- 
standing the various needs involved and the amount and kind of activity needed 
to satisf/ them. Then, better choices can be made about how the program unit 
will invest its evaluative resources. 

Most models, frameworks, or approaches to evaluation have been designed 
for a specific type of situation. Such frameworks seldom handle the variety 
•of other needs and evaluative responsibilities of local programming personnel. 
For example, a model designed to provide the social policy maker with sound 
generalizable data may have little value in helping the programmer guide a'nd 
improve a program that is in-^process, and vice versa. The kind of evaluative 
information provided by a particular evaluation approach needs to be examined 



in terms of its value compared with other evaluative information provided 
by other approaches. 

Not only must the six groups clarify and better communicate their needs 
ar,d program administrators take a more active role in trying -to establish 
talance among needs > but all six groups must take more responsibility 
for developing evaluation models. Most of the approaches included in this 
monograph — the ones that get the greatest attention in the literature —have 
been devised by outside observers, usually social scientists brought in to 
do evaluation. Though certain parts of evaluation can be contracted to 
outsiders, the users of the evaluation should maintain an active and dominant 
responsibility for determining what evaluation should do and what it should be. 

Multiple Units of Programs 

A program can be an activity carried out by one adult educator. It 
can be the total of his activities; it can be the total of what's carried 
out by everyone working for one agency or organization at one location. 
It can be the total of all like efforts carried on in several locations in 
the state or country. Evaluation is important in each concept of the program. 
But the evaluation frameworks differ. 

For example, an approach useful in evaluating a local program may not 
fit the evaluation of a national program carried on in many locations. On 
the other hand, a framework developed for the national program may not give 
enough information to guide the local program unit. 

As much as possible, relationships between evaluation policies and pro- 
cedures at various levels should be clear and the criteria consistent. The 
actual processes and techniques may need to be specific to the nature of 
the programming unit involved. There needs to be further model building 
to fit each of the concepts of program. Most of our past models come either 



from the national level or from the individual instructor's standpoint. 
Emerging models are more apt to focus on frameworks appropriate for a local 
program unit. 

Influence of State and Federal Funding 

Dependence on outside funds influences program evaluation several 
ways. First, of course, is the direct and concrete influence exerted by the 
stipulation that most federally funded programs must provide evaluative in- 
formation. Unfortunately, the funding agency often fails to help the program 
develop a meaningful evaluation. Problems arise when: 

1. Administrators of the funding agency (the men in the middle) don't 
know what they want and require evaluation hoping that ■*-he programmer 
or an outside contractor will know what they need and provide it. 

2. The funding agency has specific -cypes of evaluation it wants, but 
either is unable to communicate these specifics clearly or expects 
something completely unrealistic given the state of development of 
the field of evaluation and/or the budget restraints of the program. 

3. The funding agency is willing to let the programmer choose the type 
of evaluation investment that will be most valuable to the program, 
but fails to either communicate this latitude or to stand by earlier 
communications. Situations are compounded when funding agency per- 
sonnel change positions, responsibilities, or approaches to evalua- 
tion so rapidly that the person who okayed an evaluation plan has 
left by the time the evaluation is completed. 

In some instances, record keeping and the collection of yearly "head 
counts" seem to be done for the sake of record keeping rather than for a 
useful purpose. If at one time there was a clear rationale for data requested 
in the myriad of report forms, the rationale has been lost or it hasn't been 
clearly communicated to those supplying the information. As a result, 
annual reports and statistics sometimes bear little or an unreliable 
connection to the evaluation of programs. 

It's one thing to cope with evaluation when demands are clear and 
specific. It's another when the local level can't identify what's wanted 
and must expend energy trying to second-- guess . In such instances, considerable 



resources are spent on activities that have little value to either the 
program unit or to state or federal administrators. Often the programmer at 
the end of the chain doesn't understand what the Washington unit really 
needs. The message loses meaning as it filters down. Or^ the Washington 
unit has lost touch with the realities of the situation and asks for 
information that doesn't adequately describe the operating programs and their 
results. Devising an evaluation system that's efficient and effective for 
all concerned in a multi-location, multi-level administrative network 
has different dimensions than a system developed to work effectively foi- 
the local program. 

The need, both by those providing and those receiving funds, for a 
better understanding of evaluation an4 more accurate and efficient evaluation 
procedures triggered the expansion in ideas about evaluation that has occurred 
in the past few years. However, although progress has been made, much. still 
has to be developed and decided. Federal bureaus will have to continue to 
be partners with theorists and practioners in further refining and developing 
evaluation concepts. 

This discussion emphasizes the complexity of the evaluation needs, 
interrelationships, and importance of not trying to reduce evaluation to 
one specific procedure or to use only one evaluative approach. But federal 
funding also exerts other kindz of influences on evaluation. Influence 
comes with the nervousness among staff and clientele engendered by "soft" 
funds — funds appropriated for a set period only amd then open to review and 
reappropriation. Nervousness runs particularly high with many programs for 
the disadvantaged because openness to understanding and political positions 
of leaders have been known to vary greatly over a few-year period. 



Even initiating legislation is changed with later amendments. Programers 
faced with an urgent need to show the value of their programs also worry 
that political considerations will influence the funding, regardless of the 
presence or lack of evidence on the value of the program. Do you make a 
conscientious effort to evaluate or is it better to employ those resources 
in various forms of politicking? Certainly it^s foolhardy to ignore politics 
in the broad sense of power bases and influences • Evaluation often has 
political involvements that must be assessed and dealt with, 

Inherentness of Conflict 

Seldom have adult education programs been so closely enmeshed in conflict 
as are programs for the disadvantaged • Conflict poses additional needs for 
evaluation and heightens the trauma involved with it. Conflict is either 
inherent or imminent in a variety of ways. 

First, underlying conflict exists between the clientele and factions 
in the larger society: How representatives of both groups view the meaning 
of being disadvantaged, how each views and reacts to the suggested causes of 
that condition, and how each sees the responsibility of the other for erasing 
"disadvantage" present unceasing conflict that erupts at unexpected times and 
affects operating programs. 

There *s conflict in the use of tax monies. Some legislators, govern- 
mental officials, and groups within the public oppose the amount of money 
being spent on such programs. Others point out that the total budget 
is far less than the amount spent on defense or space exploration. Some say 
that if the federal money spent for educational and social action programs for 
disadvantaged clientele were given directly to the clientele, everyone would 
be above the poverty level. Others believe that a "dole" is incompatible 
with human dignity arid investment in human development is well worth the added cost. 



A third kind of conflict is the kind that sometimes exists between 
agencies who are competing for the same limited funds, for the attention ot 
the same clientele, and for the same image and prestige in a coinmunity . 
Considerable disagreement exists about which prograrus are most valuable. 

A program that hasn't four.d a foolproof road to success faces continuing 
conflict between programming philosophies and divergent program approaches 
and activities. For example, proponents of traditional class approaches, 
learning centers, and the problem-centered approaches may all vie for 
ascendency in adult basic education. 

Most of these conflicts involve valuing and reconciling or coping with 
opposing values. The programmer must be cible to reconcile the expectations 
of the clientele with the expectations of the agency^ with the interpretation 
of what the funded mission of the program is. Or, if unreconcilable, the 
programmer must choose from among the various expectations. At the same 
time, he must be able to successfully fend off -competitors' assessments of 
the value of his program. 

Evaluation can be an important tool in conflict situations. However, 
it must be a kind of evaluation that can deal with value questions. As yet 
this is a relatively undeveloped area in evaluation theory and in the 
systematic practice of evaluation. The scientist is trained to avoid 
valuing. Thus, his evaluation models may fail to adequately consider the 
process of dealing with values in evaluation. 

Lack of Past Operating History 

Many of the newer programs haven't been operating encugh years to: 

1. Create a generalized, positive image strong enough to provide a 
protective smoke screen and ward off demands for evidence of 
specific accomplishments. 



2. Develop the internal protective mechanisms to handle, bypass, 
or submerge evaluation issues in the same way that established 
adult education programs have, 

3. Use the wisdom of past experience in adjusting process and routine 
to i'Tiprove prograips. They're under the gun to achieve optimum 
operating effectiveness immediately and aren't given the same 
freedom of a spaxi of years to grow and develop that other adult 
education programs have had. 

4. Be able to draw on reservoir of research and experience germane to 
their endeavors; they must create as they go. 

5. Be able to draw from a pool of experience personnel. 

6. Create a positive, external image that attracts desired clientele. 
When a program lacks past operating experience, it has a particular need 

for evaluation. For example, evaluation can help programmers build an 
understanding of their efforts that might otherwise take years to identify 
through ordinary activities. Evaluation can rapidly increase the information 
available on programming and reduce the need for trial-and-error activity. 
It can be used to build an image that ordinarily comes slowly through in- 
creased contact with the program by and through the participating clientele. 
When program personnel are alert and strong enough to prevent being shackled 
with old approaches to evaluation, lack of past history provides an opportunity 
to build evaluation strategies that meet their unique situation. However, 
making use of this freedom requires strength, perseverence, and the ability 
to develop logical rationale for the evaluative activities used. 

Relationship to Established Programs 

Another influence on the new program is its relationships to the old. 
Most programs either are appended to a traditional, well-established adult 
program agency or, although standing independently, come on the scene 
much later than other social service or education programs. 
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New programs are often ambivalent in relation to the old. When there 
are conflicts, should they try to maintain a solid footing with the 
established program? Or, should they relate to the needs of their clientele 
regardless of how the "relatives" react? Should they be concerned about 
meeting the evaluative criteria accepted by the parent agency or the old 
timer in their respective field? Or, should they concern themselves only 
with the criteria of the clients and sponsors? For example, should adult 
basic education programs be judged in terms of the standards applied to 
childhood education? Or, just as materials and approaches need changing to suit 
the new clientele, do standards also need to be different? Although new 
programs sometimes feel like "orphans," often an administrative or generic tie 
to an older generation exists that affects the new program's view of evaluation. 

Assumptions About Successful Programs 

Another very real but subtle influence on evaluation is the influence 
of technological progress and managerial science on attitudes about what 
it's possible for educational programs to do. 

Because science can produce complex products in an orderly controlled 
process determined well in advance and carried through exactly as planned, 
it*s assumed similar procedures always can and should be applied to human 
beings . 

For example, we are apt to assume that: 

1. It*s always possible to plan with complete accuracy. 

2. All people need the same things. 

3. We're able to do anything we want to. 

4. We can get conclusive proof of what happened. 

5. We can always be efficient in our operations. 

6. Technological activity substitutes adequately for human activity. 



In some instances, these assumptions are well founded. However, in 
instances where they're not, the evaluation must be built in terns of 
reality rather than in false assumptions. 

Our concept of the role of a plan has changed extensively over 
the years. It has changed from one of being a guide to that of being a contract. 
It's assumed, for example, that the programmer should always be able to 
identify in advance what results his program will produce and, like the 
factory assembly line, produce those results cone what may. It's assumed 
that the programmer should by one means or another, be able to control the 
participants and other human environmental factors in the same way that man - 
can control machines. 

Many programs for disadvantaged clientele are producing greater and 
often more important side effects than the specific objectives of the pro- 
gram. One way of looking at the finding of important side effects is that 
the wrong objectives were set-^-^the programmer couldn't diagnose what the 
real needs were. However, another view is that the exact effects of a given 
program input aren't readily predictable. It may have influence (helpful or 
harmful) far beyond its designed intent. Evaluation should consider those 
results as well as how the program accomplished what it said it was going 
to do. 

Some evaluations become more an analysis of how well the program 
followed the original plan than an examination of how well the program met 
the original need. Traditional use of pre-post data collection in evaluation 
doesn't permit a program to change. Yet, the experiences of the past few 
years have shovm that change is the byword. Few programs occur exactly as 
planned and produce the results that were originally identified. In such 
cases evaluation must be able to deal ^ith questions like whether the change 



was for the better or whether it was the easy way out. When you're evaluating 
a program rather than a plan, the plan should be treated as a means to the 
desired end, and the evaluation should emphasize whether that end was attained. 

Mass production leads even knowledgeable adult educators who espouse 
philosophies of individual difference into the trap of expecting all partici- 
pants to achieve the same things. Whether this is good or bad depends on 
whether those things are really important to all of the participants and 
the extent of varying needs within the group. When participants are extremely 
heterogenous in needs and interest, evaluation procedures should recognize and 
deal with individual differences. Success then would be determined by the 
number of people helped to do something important to them rather than by . .e 
average gain of the whole b^'^o^P* 

Another influence business and technology fosters is a feeling of 
invincibility and unrealiatic belief in our ability. Standards for success 
are often set in terms of the need or of a "pie-ln-the-sky" hope of success. 
They assume that we not only can produce chat pie, but produce it within 
one year. We either have too little accummulated wisdom to set realistic 
rather than idealistic standards for t'.uccess or we ignore what we do know 
because we hope we can do more. We expect miracles — too few resources 
invested for too short a time to maka mammoth strides in changing behavior. 
Goals that go beyond what we're sure we can accomplish do stimulate effort, 
but may result in unrealistic criticism when they're not completely attained. 

Although efficiency is a quality to work towar-d, we may have little 
concept of what constitutes efficient education. With few exceptions, dis- 
advantaged adults aren't an efficient clientele to help make major gains. 
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Although some are immediately able and highly motivated, many have suffered 
school and nonschool experiences that have reduced their faith in their 
ability to learn and interest in learning. The kind of input needed to 
establish trust and increase motivation is much greater than in many other 
programs. In addition, many of the clients face crisis interruptions that 
are more impo^^tant than program participation. So, attendance is often 
irregular and dropouts frequent for reasons other than disliking the program. 
Evaluation is needed: (l) to determine what we can realistically expect 
to occur from programs with this clientele and (2) to find the best processes 
for being of greatest help. 

As we transfer beliefs about success from the field of technology, we 
often forget all the years of work that went into the laboratory and field 
research that perfected the present assembly line procedures; we expect 
programs to produce the first time off the drawing board. 

We may be misled, too, into believing evaluation can do everjrthing we 
would like it to do. Just as our standards for program success may be un- 
realistic, so our standards for efficacy of evaluation may be equally un-- 
realistic. Although we want conclusive and indisputable proof of accomplishments, 
how great is the chance of getting completely firm data when programs operate 
in a real and dynamic world where it's not possible to isolate participants 
and thus be able to study the effects of the program in isolation? 

Making laboratory type research realistic in a living, uncontrollable 
world is one of the headaches of evaluation. In coping with that headache, how- 
ever, we sometimes forget more in5>ortant things; for example, the number of 
laboratory experiments that were made before data were accepted as proof of 
a scientific finding. If the resources needed for repetition of evaluation 
and the many analyses needed to develop even fairly conclusive proof were 



actually expended, evaluation would cost far more than actual program 
operation. In addition, even though we amass results data on a given pro- 
gram, in how many instances do we have an adequate "season record" to be 
able to interpret the degree of success of a particular operation? Yet, 
we usually treat data not as input into the season record determining 
player averages, but as understandable and useable in isolation from 
comparative data on other programming operations. 

In searching for objectivity, we may have been misled by the role of 
calculators and computers. Rather than considering the data as ways of 
helping human judgment be more objective, we sometimes think data constitute 
evaluation and accept, without careful scrutiny, evaluations with impressive 
data treated by highly sophisticated statistical processes. This trans- 
ference means there's a tendency to define evaluation as the data involved 
in the activity rather than the conclusions of the evaluation or the process 
by which conclusions are reached. 

We transfer some things from our technological successes, but we 
sometimes fail to transfer others. For example, the quality found when the 
product is evaluated at the end oJ: production depends on careful evaluation of 
it at each step in the process. Yet some concepts of program evaluation 
ignore the importance of evaluating decisions throughout the total sequence 
of the program — ^for example, evaluating the purpose and objectives to see 
if they're realistic and attainable. 

In some situations, most of our assumptions should be applied to pro- 
grams. But, in others, they may be completely unrealistic. Concepts and 
strategies of evaluation must be founded on assumptions that are accurate 
within the environment of the evaluation. 
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These seven influences are only a few of the real-world conditions that 
affect evaluation of programs for disadvantaged adults and make it confusing 
and frustrating. The kind of evaluation models that are most needed are 
those that can deal with real-life situations and make valuable contribu- 
tions in an everyday environment. Evaluation has to be realistic in terms 
of the real operational context of the program and its environment without 
becoming biased or distorted by real conditions. Recent experiences have 
caused evaluation to expand and change. 

Contemporary Ideas About Program Evaluation 

Ideas about evaluation are changing. Beliefs about program evaluation 
plateaued for a few years during the 1950s and 1960s when evaluation was 
equated with research methodology to such an extent that sometimes the terms 
measurement and evaluation were treated interchangeably. During that period, 
too, evaluation was often limited to determining whether content-specific 
objectives had been achieved. 

Then the late 1960s brought an influx of new programs and new demands for 
eva].uation. Established concepts didn*t deliver. As a result, new ideas 
about evaluation emerged and new frameworks appeared • There *s considerable 
divergence in those ideas. Most of them are still in the trial-and-testing 
stage. Many paths are being taken off the plateau of the earlier period, but 
few of those paths are widely accepted. None can be considered the main route. 
Some explore evaluation from the standpoint of its purpose, some from the stand 
point of need, some from the view of organization and system, and some from 
the interactive elements involved. New definitions of evaluation are evolving 
(see Table 1). 

Although the needs and pressures outlined earlier have influenced the 
changes occurring in evaluation approaches, development of theory and 
practice is not yet sufficient to completely deal with ciirrent needs* 



Table 1 

CONTEMPORARY DEFINITIONS OF EVALUATION 



Evaluation is the process of 
ascertaining the decision areas of 
concern, selecting appropriate in- 
formation, and collecting and an- 
alyzing information in order to 
report summary data useful to de- 
cision makers in selecting among 
alternatives. 

Alkin, Marvin C. "Evalua- 
tion Theory Development." 
Evaluation Comment^ II 
(October, 1969). 



Evaluation is the systematic 
process of judging the worth, de- 
sirability, effectiveness, or 
adequacy of something according to 
definite criteria and purposes. 
The judgment is based upon a care- 
ful comparison of observation data 
with criteria standards. Precise 
definitions of what is to be ap- 
praised, clearly stated purposes, 
specific standards for the criteria 
traits, accurate observations and 
measurements, and logical conclu- 
sions are the 'hallmarks of valid 
evaluation. 

Harris, Wilbur. "The Nature 
and Functions of Educational 
Evaluations." Peabody Joidr- 
nal of Education^ XLVl 
(September, 1968), 95. 
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Evaluation is quality control 
of the processes and outcomes of an 
educational program. 

Gottman, John M., and Robert 
Clasen. Evaluation in Educa^ 
tion: A Praationer^e Guide. 
Itasca, Illinois J P» E. Pea- 
cock Publishers, Inc., 1972, p. 16. 



Evaluation is the determination 
(whether based on opinions, records, 
subjective or objective data) of the 
results (whether desirable or undesir- 
able; transient or permanent; immediate 
or delayed) attained by some activity 
(whether a program or part of a pro- 
gram . . . , an ongoing or one-shot 
approach) designed to accomplish some 
valued goal or objective (whether 
ultimate, intermediate, or immediate, 
effort or performance, long or short 
range). This definition contains four 
key dimensions: (l) process — the 
"determination"; (2) criteria—the 
"results"; (3) stimulus—the "activity"; 
and (4) value— the "objective." The 
scientific method with its accompany- 
ing research techniques then provides 
the most promising means for "deter- 
mining" the relationship of the 
"stimulus" to the "objective" in terms 
of ' measurable "criteria." 

Suchman, Edward A. Evaluative 
Research* New York: Russell 
Sage Foundation, 1967, pp. 31-32. 

oooaooo o ooo o oooooooooooooop eoQQQg 

Social program evaluation is the 
systematic accumulation of facts for 
providing information about the achieve- 
ment of program requisites and goals 
relative to efforts, effectiveness, and 
efficiency within any stage of program 
development. The factors of evalua- 
tion may be obtained through a variety 
of relatively systematic techniques, 
and they are incorporated into some 
designated system of values for making 
decisions about social program. 

Tripodi, Tony, Phillip Fellin, 
and Irwin Epstein. Social 
Progtwn Evaluation. Itasca, 
Illinois: F- E. Peacock 
Publishers, Inc., 1971, p. 12. 
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However, the key ideas that are emerging can help you develop strategies 
that will work in your situation. Among the more valuable new ideas about 
evaluation are the following: 

1. Pr»ograni evaluation is a process rather than a procedure. It's 
generic rather than specific, 

2. Program evaluation is more than examining the attainment of objectives. 

3. Program evaluation is more than just evaluating the resiats of a 
program. 

4. Program evaluation is more than instructional evaluation. 

5. Program evaluation is different from evaluative research and 
program, research. 

6. Program evaluation is a management tool. 

7. Program evaluation is people centered. 

Program Evaluation Is a Process Rather Than a Procedure. 
It Is Generic Rather Than Specific 

The definitions in Tabl,e 1 are much broader than our old ones* They 
recognize thaL one specific procedural definition doesn't meet the range of 
needs or fully use the powerful potential of evaluation. Instead evaluation 
is most easily handled and has greatest utility if it's considered a generic 
term and used as such. Thus, program evaluation encompasses one or more 
generalizable processes, many evaluative strategies and approaches — more 
than 50 have emerged in the last 6 years, and specific evaluative activities. 

The two most prevalent ways of looking at evaluation as a generalizable 
process are: 

1. As a process of forming judgments about programs using criteria 
or standards of comparison and descriptions of what occurred and 
resulted in the program. 

2. As a process of using information in comparing alternatives in 
reaching program decisions. 

The first concept places the emphasis on judging and forming conclusions 

about program activities and program resiilts. These conclusions are then fed 
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into decisions about further and future activity. The second concept 
emphasizes identifying alternatives and then using evaluation to help choose 
among those alternatives. The first concept would deal with a question like, 
"How important is this program?" The second would explore, "Which of 
these two programs is more important?" Or using another example, the first 
approach would examine how effective a given approach is in carrying out 
the agency's mission. The second would compare two or more approaches to 
see which is more effective. 

There's a good deal of similarity between the two concepts of evaluation 
as a generalizable process. Both are so new that little attention has yet 
been given to how they relate to each other— how judgment and decision 
making function in relation to each other. . Much more exploration of the 
linkages between the two is needed. 

Accepting program evaluation as a generic term rather than restricting 
it to one uniform definition means that those requiring or proposing evaluation 
will have to be more exact in describing the purpose, processes, and procedures 
that they're referring to when they use the term "evaluation." Greater precision 
should mean greater clarity and more opportunity to fit strategy and approach 
to the existing environment. 

Program Evaluation Is More Than 
Examining the Attainment of Objectives 

A few years ago when evaluation was suggested, we automatically brought 

out a framework that examined the extent to which objectives had been met. 

This, of course, is a fairly important evaluation to make of a program. 

But, it's not the sole, and sometimes may not even be the most important, 

analysis of results. Two of the major emerging developments contributing 

to a better understanding of evaluating results are: (l) the expansion of 



what needs to be assessed when you^re concerned with examining the results 
of a program and (2) the distinction between describing and evaluating 
results . 

More Than Objectives. Evaluation is beginning to look beyond objectives. 
The growing number of results approaches that go beyond the results stated 
in the objectives aren't ignoring objectives nor discounting their role in 
guiding programs and bringing results into better focus. They are, however, 
recognizing that in the complexities that exist in most programming situations, 
programs are apt to produce unanticipated results. Sometimes the values or 
harms of these unanticipated results are more important than the objectives 
per se. For example, with some clientele, the fact that they're in social 
contact with others in the program may have value far and beyond what they 
actually learn from the program. In some instances, programs that show few 
results in terms of their content-specific objectives may still be making 
extremely valuable contributions to the participants and society. 

Harmful results also need to be considered. For example, a person may 
learn the multiplication tables, but learn to hate arithmetic at the same 
time. A person may meet the objectives of a program very well, but be so 
disconcerted by it that he won't return to it or similar educational 
experiences . 

Many of the broader approaches coii5>are the amount of results produced 
to: (1) the needs that initated the program, (2) the kind and amount of 
results that must be produced if the agency is to attain its mission, 
(3) broad competency standards, or (4) statements of the kind of results that 
a program conducted with the particular clientele using a certain amount of 
input should be expected to produce. Objectives are symbols or transmittal 
links to one or more of these broader definitions of results such as * 
meeting a need, or carrying out a mission. If objectives are well chosen 
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and directly on target, results judged against the broader base will also 
show that the objectives have been attained. However, if the objectives 
aren*t germane, their attainment doesn*t necessarily mean that any of the 
four broader bases have been achieved. In many instances, objectives 
become so narrow or so side-tracked that little is accomplished through 
them. An evaluation issue is: Should programs only be examined in terms 
of what the programmer thinks he*s doing (that is, what he has defined as his 
object .)? Or should they be examined in terms of what he should be doing 
(that is, what should be happening if needs are being met, mission accomplished, 
broad competency standards attained, regardless of what his objectives were)? 

Evaluation that's concerned with the overall effectiveness of a 
program is concerned not only with results in terms of behavioral changes in 
people, but also with the proportion of the potential clientele that's 
i reached, the balance in types of people reached, the extent to which the^ 

j results deal with urgent and continual need, and- the care with which partici- 

I pant, agency, and societal resources are tised. Program evaluation is as 

t 

\ concerned about the value and suitability of the program as it is with 

1 whether its purpose is accomplished. 

\ Description Vs. Evaluation. The second major step in improving 

understanding of what's being done when a pTOgram is evaluated in terms 
of its results is the need to distinguish between deBoribing the results 
of a program — providing evidence of what results did occur — and evaluating 
those results. Past approaches to result evaluation limited themselves 
primarily to identi:fying if results did occur and testing whether they 
probably came about as a result of the program. Seldom did they try to 
identify the sufficiency of those results. A statistically significant 
chi-square value in a comparison between program participants and a control 
group was considered enough, even though half of the participants hadn't 
I achieved the expected results. 

ERLC 
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Evaluating results gets into questions requiring more information than 
just whether the program produced results* It involves such sticky questions 
as: Are the results consistent with what's logical to expect? Are the 
results important? Do they contribute more to the participants and society 
than if the time and other resources had been invested in other things? 
Were they produced at a reasonable cost? Are the results sufficient in terms 
of the overall need? Are they sufficient in the expectations of the par- 
ticipants and the amount of time and energy they invested? Is there any 
evidence that it's realistic to expect a program to produce more results than 
this one has, given the same budget, personnel, and working conditions? 
If the results are insufficient, does it mean that the program isn't effective 
or that changes need to be made in the way the program's carried out? Have 
The results been replicated with enough consistent findings to clearly decide 
on the value of this particular program? Are the results of one program 
significantly more valuable than those that could be produced throxigh an 
alternative program with the same expenditure of resources? 

Most of these questions can't be answered by traditional evaluation 
approaches. Ways of dealing with these types of questions still must be 
developed. 

Program Evaluation Is More Than 
Evaluating the Results of a Program 

One of the most important of the recent developments is the distinguishing 

of the evalxjation of program results from the evaluation of programs. For 

years, evaluation has been thought of aa primarily a summative activity in 

which the results of a program were studied. Now formative* evaluation — 

evalxiation that influences that program while it's progressing— is gaining 



27 



equal importance. Evaluation of each of the key processes involved in programming 
(planning, management, interaction, etc.) and of each of the key stages or 
activities is essential to achieving results. 

Evaluating the extent of need for various types of programs and using 
evaluation in setting priorities is imperative. Evaluating the outcomes 
expected of a program to see if they're, realistic and focus on the most 
crucial needs, evaluating the p:..ans for the program to see if adequate inputs 
are being marshalled and if the activities planned can produce the type of 
results expected, and monitoring the processes involved to see that they 
stay on target and that repetition or reinforcement are added to the original 
plan if such activity is necessary-.-^ll are essential if a program is to 
achieve maximum results. Evaluation that influences ongoing develop- 
ments in the program has great value — it improves and gives immediate 
benefits . 

More is involved here than the usual categorization in terms of process 
versus product evaluation or as some people say, the evaluation of means and 
the evaluation of attainment of ends. Program evaluation as a generic term 
can include the evaluative activities that focus on either process or product. 
But the emerging emphasis is on frameworks that consider both process and 
prodtict and, more importantly, the interrelatedness of the two. Program 
evaluation deals with the program as a functioning, producing system. It 
includes results as one key component, but also examines other aspects of 
the program. Processes and structural components such as taacher's performance, 
materials, and facilities are examined in relation to how they can be more 
effective in generating a viable, valuable program. The emerging models 
differ in how they define the parts of the functioning system. Some program 
evaluation approaches focus on processes, others on program components. 
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Program evaluation serves both during the actual program operation and in 
retrospectively analyzing how the particular level of results was produced. 
At a broader level, it expands our knowledge of what makes up successful 
programs under specific environmental conditions (backgrounds of students, 
extent of budgetary support, social pressures). Such models are particularly 
important to new and/or controversial programs. 

This broader track of program evaluation emphasizes developing and im- 
proving programs. Approaches that focus only on results are primarily con- 
cerned with questions about the termination or continuance .of the program. 
Both are essential. One doesn't substitute for the other. Every program 
should be reviewed periodically to see if it should be continued. (Old programs 
never die. They don't even fade away.) But when new programs are developing 
and when decisions have been made that established programs will continue, 
program evaluation that emphasizes improvemont is crucial. 

Program Evaluation Is More 
Than Instructional Evaluation 

Until recently, "program" has been a little used word in educational 
circles. Few school personnel felt a need for any terms beyond curriculum 
and instruction. The Tyler approach in the 1950s clarified the relationship 
between evaluating students and evaluating instruction. It used objectives 
as the focal point to guide the data gathering on student performance. 

Adult education began equating the term program evaluation with the 
Tyler construct in general education that was primarily concerned with the 
evaluation of instruction. Thus, for some time program evaluation was seen 
as being how well participants achieved the instructional objectives set 
for the group. Now the distinctions between evaluation of instruction and 
program evaluation are becoming clearer. 
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Instructional evaluation is concerned primarily with the specific course 
or program activity. Program evaluation is concerned with the additive effects 
of a series of instructional components . What are the combined results of the 
courses taken in a two-year apprentice program? Instructional evaluation is 
usually most concerned with knowledge, skills, and attitude change. Program 
evaluation is concerned with the impact that those changes and program 
participation have on the person and those he»s in contact with. Instructional 
evaluation is more apt to deal with how the program satisfies the specific 
needs of individual learners; program evaluation is more apt to deal with how 
the program meets the needs of the community or a subsection of society. 

Program evaluation includes instructional evaluation, but deals with addi- 
tional things. It*s concerned, for example, with establishing priorities 
among instructional programs. It*s concerned with the extent to which the 
instruction is attaining the institutional goals of the agency. As an .ex- 
ample, if the agency (institution) holds a broad objective of helping adults 
develop ability to think analytically and creatively, how well is such an 
objective being accomplished within a specific course or program activity? 

Program evaluation is concerned with whether adequate resources are being 
applied to the right programs. It deals with such issues as whether to be all 
things to all people, or whether to concentrate resources in certain priority 
thrusts. Instructional evaluation is concerned with the quality and results 
of specific activities within the total program. Program evaluation is con- 
cerned with the extent to which the agency is carrying out its mission. In- 
structional evaluation controls and improves the specific programmer-participant 
relationship. 

Program evaluation doesn^t substitute for instructional evaluation; and 
instructional evaluation doesn't substitute for program evaluation. Even 
though good instructional evaluation provides essential information for evaluating 
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a program, it doesn't provide all of the information that's necessary even in 
pror.r.ir.s where instruction makes up the total of the program. It's possible 
to do a good job of instructional evaluation without ever really doing program 
evaluation. 

Program Evaluation Is Different from 
Evaluative Research and Program Research 

Considerable debate is currently going on about whether evaluation and 
research are the same thing. There seems to be three related activities that 
use similar methods. All are important in programming. 

Program reseccrch looks for new and generalizable knowledge. It expands 
our understanding of programming. Findings are useful in a variety of settings. 
It's trying to formulate theories and principles. Occasionally program 
evaluation provides a context for program research or program research uses 
program evaluation as one of its parts. In these cases » however, -benefits 
to the program being evaluated are secondary. The focus is on an audience way 
beyond those involved in the program used for the research. 

Evaluative research is a specialized branch of research that deals 
with valuing. It*s concerned with identi^ing the value of programs or with com 
paring the values of two or more programs. It*s concerned with whether the 
type of program can produce something of value in relation to a need. It 
differs from program evaluation in that, as a form of research, it*s interested 
in these questions only in the broad context of generalizable answers, 
answers that contribute new knowledge and lead to theory and principles. The 
contribution to the particular program examined is secondary. Evaluative 
research may be an important procedure in program evaluation at a national 
level or in new programming situations. 

Program evaluations as distinguished from research, usuall/ deals with old 
questions about programs. It*s not trying to get generalizable information — it' 
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trying to get sound and reliable information to use within the context of 
the specific program. Program evaluation is a field in its own right. It 
resembles itr> research cousins ♦ but there's a growing awareness that it 
shouldn't be treated as a twin. Its processes and procedures need to be 
developed within its own context rather than borrowed directly from other 
parts of the family. 

The confusion of evaluation and research has led to underdevelopment 
of evaluation procedures in at least five main areas: suiting data techniques 
to evaluative needs, understanding criteria, distinguishing between descrip- 
tion and judgment, providing useable infomnation, and communicating evaluative 
findings . 

First, although evaluation may borrow data treatment methods like 
precise instrumentation and statistical processes from research, they 
must be tested for validity and reliability as tools. in evaluation rather 
than automaticaJly transferred and applied as they'd be in research situations. 
Conside^/cJble adaptation may be needed. For example, when is it best to 
examine average gain for a whole group and when is it better to examine the 
number of students making specific amounts of gain? Is it necessary to rule 
out chance occurrence at the .05 level, or would a .*^0 or .25 level be 
adequate in evaluation? Much of test development is based on procedures for- 
discriminating among students to assign grades in line with a normal curve. 
In evaluation, such discrimination is i? relevant. The type of data recorded 
on student-^teacher interactions might take a different fonri,, when a program 
is being evaluated than it does when it will be us^d to test- a particular research 
hypothesis. A good many other examples could be given. 

Although we're becoming aware that new procedures are needed and standard 
research techniques may need to be amended and adapted, this is a relatively new 
field. Some evaluators and researchers, for example, have been suiting the 
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statistical significance level— the level at which chance occurrence is 
ruled out — ^to the context of the problem foi^ some years now. But little 
has been done to identify the conditions under which a .25 or a .10 level may 
be appropriate in program evaluation. In many instances, evaluation may not 
require the same depree of stringency in data dealing as does research. This 
may be particularly true when the consequences of inaccuracy are relatively 
small. 

Second, criteria must receive much more attent^ »n in evaluation than 
in research. In research, hypotheses serve as the basis for deciding what 
information to gather and how it*s to be organized to be interpreted. In 
evaluation, criteria — decisions about what the program will be judged on — play 
the same role. Evaluative criteria aren*t always the same ones that applj- 
in research, although there's a similarity in the role of criteria in the two 
situations. Unfortunately, most research texts assume criteria without 
dealing with them as important phenomena. The 'fledging researcher has 
often been so enthralled with the data-handling procedures that he hasn't 
mastered the concept of criteria as it applies in research. So, he's unable 
to carry even the basic concept into evaluation. 

Although somh program c;t»iteria presently exist either in a formal or 
subconscious state, they must be probed for clarity, reality, and the 
extent to which they portray the most valuable things in programs and programming. 
Programs can appropriately be judged on a wide range of criteria, some of 
which are in conflict with others. Criteria need to be ordered in terms of 
their importance. For example, which program is more successful-- the one 
that reaches a large number of psople with a moderate amount of help to them, or 
the one that gives great help to a very small number? Which program is more 
successful — the one producing efficient results for a short period of time, or 
the one with only moderate results with a longer retention time? 
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Criteria describe what's of value or good. They're debatable. Although 
programmers sometimes wish to avoid the trauma of debate, a confrontation 
on criteria can clear up some of the ambiguities and frustrations that plague 
evaluation. 

Program personnel and their professional gioups are responsible for 
developing, screening, and debating criteria. In doing so, considerable 
attention should be given to the views of clientele. Although experts 
knowledgeable in a given field can make recommendations and researchers can 
help clarify issues and determine the degree to which certain criteria are 
commonly held, it's not the outsider's prerogative to establish the criteria 
for the evaluation. 

A third major area for development is the greater understanding of the 
difference between description and evaluation. A presentation of research 
type findings about a program usually constitutes a description of the, 
program or of its results. Evaluation doesn't ' really occur unless judgments are made 
about such things as whether the processes used were suitable and of high quality or 
whether the results produced were enough for the resources expended, or how 
effective the processes were in producing the results • Research tries hard 
4*ct to be judgmental. Evaluation, on the other hand, doesn't fui '"r 
role unless it takes responsibility for delivering conclusions that 
judge, identify the adequacy of what's being described, or show relative 
advantages or disadvantages of two or more alternatives. 

It's probable that in some situations, particularly when reporting to 
outsiders, the programmer may only want to describe what happened in the 
program and /or* its results. However, in decision making, descriptions are 
useless unless judgments are made and values assigned* Unfortunately, 
sometimes adult educators who have majored in science fields and had graduate 



courses in research methodology not only haven't had experiences that 
help them improve their ability to judge programs, but also have been 
taught to mistrust and avoid situations where value judgments are involved. 

A fourth major area for development deals with the importance of being 
able to use data. It*s not enough to just collect accurate data. Those 
data must be accurately and immediately interpreted within a specific context. 
Interpretation rests with man. Wisdom and experience in forming and using 
criteria, in assessing the limitations of data and the potential consequence 
those limitations pose, and finally in making, communicating, and defending 
judgments are more important skills in evaluation than are the skills of 
data gathering and statistical analysis. Scientifically produced data 
are a valuable input in evaluation, but seldom should stand alone as the 
output from it. 

From one perspective, data-gathering activities are a separate function 
from evaluation. They're essential to it, but can be considered as prior 
and prerequisite to the actual act of evaluation. Unfortunately, as we've 
borrowed from research, we've been more apt to focus on the data-gathering 
techniques than we have on the part of research dealing with using data. 
We've hoped that having high quality data would solve our problems. We're 
finding that data in and of themselves give few answers. The next step in 
understanding evaluation and its procedural development must deal with 
processes of the mind that are essential in using information in evaluating 
programs . 

Fifth, our associating evaluation with research sometimes leads to 
unproductive ways of communicating results. We may be too prone to 
prepare reports in traditional research ways that relegate conclusions and 
implications to the end of the report. The people who should use evaluation 
seldom get to the end of the reports. If they do, they are often so 
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confused by academic hedging and research jargon that they can't decipher 
what the evaluation really means in terms of their own needs and situation* 
Currently a variety of ways of inserting evaluation findings and conclusions 
into the bloodstream of the program and conununity are being emphasized. 

Meaningful evaluation has been stunted by failure to recognize that 
program evaluation and evaluative research, although related, aren't the 
same. Now that the nature of the differences are clearer, progress should 
be made in developing specific procedures that work well in evaluation. 

Program Evaluation Is a Management Tool 

Evaluation can be a powerful working tool in programming. It's not an 
end in itself, something to engage in because it's intrinsically good, but 
a way to get things done. With the clearer understanding of evaluation's 
role in decision making, comes a better understanding of the value of 
evaluation in guiding and managing program activities. Evaluation provides 
a basis for better program choices and for more rapid response to needs 
for improvement. It can be a tool in improving total operating efficiency 
as well as providing the client and society with more effective programs. 

Distinguishing between evaluation and research ejnd clarifying the role of 

I 

research-type data in evaluation, paves the way j for a clearer understanding 
of evaluation as a natural human process-*-^ continuous programming activity 
rather than , an episodic or extensive but infrequent effort. The emphasis 
then is on improving accuracy of this process by introducing more system 
into the way it's carried out rather than by replacing it completely with 
periodic, systematic, formal evaluations. Evaluation in terms of intuitive 
judgments does and should go on continuously. It should be an automatic 
part of an adult educator's professional skills. He should form the basis 
for making accurate judgments as a part of gaining professional competence, in 
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the same way a doctor makes accurate diagnoses. Just as the doctor uses 
lab tests when he feels they're warranted, so the adult educator should know 
when more extensive, systematic processes are needed to improve the sound- 
ness of his judgment and when systematic processes are worth their cost. 
Both natural and systematic evaluation play important roles; neither re- 
places the other. Both are essential in managing programs. 

Evaluation has to be well managed if it's to serve as a management tool. 
The degree of time invested in evaluation, degree of formality, and objectivity 
of procedures should be matched with the degree of value the' particular 
exploration can produce. Syste;natic evaluation is a costly activity. The 
degree of system used should be equivalent to the importance of the results 
produced. Some situations require the best possible criteria and most 
scientifically pure data' it's possible to get, regardless of the cost in 
time, money, or inconvenience to participants. In other instances," following 
the usual systematic procedures will require too great an investment for 
the importance of the judgments to be made . Attention must also be given to 
what information needs to be instantly available in an up-to-date form. The 
typical point in time, or episodic, research-^like effort often doesn't produce 
results in time to influence ongoing decisions. One of the challenges in 
designing the information systems that provide input into continuous evaluation 
and decision making is reconciling the cost of keeping information continuously 
up to date with the extensiveness and value of its use. 

The overall evaluation strategy of an agency or programmer should give 
equal attention to the everyday processes of evaluation, "diagnostic" 
evaluation, and more systematic endeavors as requi red. Ev aluation should be 
designed for the context in which it will be used. What questions must be 
answered or decisions made? By whom? By what date? A clear understanding 
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of why you •re evaluating and what you want to accomplish by that evaluation 
is essential in effectively using evaluation as a management tool. 

Program Evaluation Is People Centered 

Another major development in evaluation is the increased emphasis 
on evaluation's relationships to people • Evaluation as input into decision 
making emphasizes the need of interface and interaction. Evaluation for 
program Improvement recognizes that those who must make the improvements 
must be actively involved in the evaluation. Recognition of close relationships 
between evaluation and politics and policy emphasizes the human element. The 
growing awareness of the central place of criteria and judgment's in evaluation 
emphasizes that those criteria come from people and that judgment is made 
by people. Even in terms of data, the need for involving a variety of 
people in interpreting data so that a more complete picture can be secured is 
becoming more apparent* 

For a few years we tried to take evaluation out of the people realm, 
hoping to increase its objectivity* Lately, however, there's a growing return 
to active involvement of many minds as conclusions are formulated and 
decisions made. We need ways of increasing objectivity while relating closely 
to the people involved during the evaluation process. 

More responsibility for evaluation is resting with program personnel. 
The role of outside evaluation consultants is changing • In addition to helping 
the system cope with data, the consultant must be able to serve as a catalyst 
and help program personnel understand and use evaluation findings. He must 
be able to help the system design evaluation that will be useful and meaningful 
to it. This may be different from the traditional assignment of designing 
a research approach to examine results. 
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Perhaps its important to caution you that you've just been riding the 
vanguard of the thinkihg about evaluation. These seven ideas have not been 
widely accepted yet. However, they hold promise for making evaluation a 
much more valuable force in programming. 

The discussion of contemporary ideas has been general and hasn't 

related specifically to programs for disadvantaged adults* What do these 

changes mean in terms of such programs? Here are a few implications. 

1. Pressure for evidence of results is great. However, even though 
much energy is going into this type of evaluation, equal attention 
should be paid to evaluation that helps programs develop and im- 
prove. Many programs for disadvantaged adults are really in very 
preliminary stages of development-^-the search and error 
procedural refinement stage — where good formative evaluation is 
crucial . 



2. A second look should be taken at che traditional ways of using 
and interpreting research type data on results. Are the standards 
being used realistic or are they laboratory standards and not 
life geared? 

3. Those programs where past experience is limited or controversy is 
inherent may want to be sure to examine a wider range of results 
than just those specified in the objectives. 

^. At least two types of evaluation are needed in most local programs 
for the disadvantaged. One monitors instruction (or other type 
of programmer-^articipant interaction) to provide feedback to the 
participant and to help the programmer make decisions as the inter- 
action continues. The other deals with broader program questions, 
such as are the right people being reached and the most efficient 
approach being used? 

5. Because of the urgent need for becoming effective and efficient within 
a rapid time span, evaluation that facilitates better program manage- 
ment is extremely important in programs for disadvantaged. 

6. A people-centered approach to evaluation that involves both staff 
and clientele in determining what will be evaluated, what criteria 
will be applied, how data will be interpreted, and in forming 
judgments and recommendations is particularly important in programs 
for the disadvantaged. 

7. Finally, perhaps the most important implication is that which comes 
from accepting evaluation as a generic term including a range of 
strategies and approaches. Programs for the disadvantaged will 

get the most mileage out of evaluation when the approach is suited to 
the need and the programmer or agency has a well-stocked kit of 
evaluation tools. 
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We've now set the context for examining some of these evaluative tools. 
Current Approaches to Evaluation 

The last section of thif ""^nograph summarizes more than 50 approaches 
to evaluation. The terms "approach," "model," and "framework" are used inter- 
changeably. Some are preliminary sketches; others are extensively developed. 
Some are adapted from processes that were used; others are designed as general 
ways of conceptualizing evaluation. Few of the authors would consider their 
materials "models" in either the theoretical or the "ideal" sense, yet all 
provide ideas to the programmer in helping him set evaluation strategy. 

Even though this m^aograph includes a sizeable number of summaries 
from a variety of social fields, it isn't exhaustive. New models are 
constantly emerging. Yet, there's need for considerably more work — particularly 
on the part of those vitally affected by the evaluation. 

The reader that finds the number appalling will probably ask: "Why so 
many?" "How do they relate to each other?" "How can I sort them out?" "Which 
should I pay attention to?" 

We'll talk for a moment about the first two questions. Section II 
is designed to tackle the third and fourth. 

In part, the proliferation of approaches represents a groping and a 
search for relevant models that was triggered when the established approaches 
to evaluation were found wanting. The ntunber of approaches may decrease as 
acceptance and new stabilization occurs. On the other hand, the proliferation 
also represents the awareness that many kinds of evaluation are needed. 

The situation can be likened to examining a mountain. The geographer 
is interested in topography. The geologist in rock samples. The mountain 
climber, the engineer designing a railroad, the pilot flying over the 
mountains, and the native going from one valley to another are all getting 



data about the mountain range to try and describe, evaluate and deal with it. 
But, the data gotten and the approach to getting that information are far from 
the same* 

In many ways programs are like a mountain. They're complex with many 
planes and facets. These planes and facets can be examined in a variety of 
ways. People have varying needs and purposes in relation to describing and 
evaluating theia. Many types of information exist that are useful to 
those purposes. Therefore, just as there are many ways in which a mountain 
can be described, analyzed, and evaluated, so are there many ways that pro- 
grams can be examined and evaluated. The approach you take depends on your 
particular needs. 

For centuries, people thought they knew what mountains were. Then art 
and science provided new tools of perception and analysis and our understand- 
ing increased many-^old. We're moving toward that stage in program evaluation, 
but as yet it's more a random, searching exploration than one clearly 
governed by distinctly different approaches. We're at the pretaxcnomic stage. 
Little has been done in categorizing approaches. But from very rough pre- 
liminary classifications like that developed in this section, you're better 
able to identiJ^ central routes of approach and gaps. 

The analogy leaves us vulnerable to the questions, "Are we making 
mountains out of molehills? Must program evaluation be this complex?" 
Perhaps one of our problems has been that we've tried to treat programs 
like molehills and to use only one analytic approach to them. By recognizing 
their complex and dynamic topography and by accepting evaluation as a com- 
plex set of tools from which appropriate ones are selected to fit the Aspect 
that is being examined, we may actually make evaluation simpler and less 
frustrating. 
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But why include so many approaches in a reference designed for the 
practioner? Several alternatives emerged. (1) Ve could limit the 
monograph to the approaches that are roost fully developed as models in the 
academic sense; (2) select prototypes or examples of different forms of program 
evaluation and let them serve as representative of the others; (3) narrow 
the definition so that only one stibgroup of approaches is included — for example, 
only those drawn from education or even more specifically those developed 
for adult education and programs for the disadvantaged; or (^) present as 
full an assortment as possible, but provide some general type of grouping 
for ease in sorting and selecting approaches. 

We chose the latter alternative for several reasons* First, few of 
the frameworks are sufficiently developed and tested to stand out as models 
for the field. 

Second, although similarities exist, each approach has at least 
one unique idea that sets it apart from its cousins. The summation of 
these ideas may be of more value than any one approach as a separate entity* 

Third, although the materials developed for adult education are ex- 
tremely useful, they don*t as yet provide as wide a set of choices as do 
other fields. Although soiorces could have been limited to education, some 
adult education programs for the disadvantaged are much broader than pro- 
grams developed within formal education and are more like those of health 
and social services. Therefore, a few approaches from other social fields 
have been included. 

Fourth, including a fuller range increases your perspective of program 
evaluation as a generic activity and your understanding of the range of ap- 
proaches that are useful. 

Fifth, you may be interested in how the models and approaches you*re 
familiar with relate to one another and to the broad field. 
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Finally, and most important, there's no guarantee that one approach will 
be more useful to you than another. Ten readers are apt to be caught by 
10 different approaches.' What has value in one situation may have little 
value in another. 

The next question is: If we're not restricting the number, how do we 
organize them? The mind can't retain and deal with more than a few items at 
a time. Therefore, how can the various approaches be sorted and classified 
to help you grasp them? Again several alternatives appeared: (1) sorting 
in a technical, taxonomic form; (2) sorting by source of origin; (3) sorting 
by extent to which approaches are commonly known or (4) developing some 
general groupings that may be useful to the practioner. We chose the 
last alternative. However, preliminary efforts in regard to the first three 
systems of organization are included in the appendix if you're particularly 
interested in classifying evaluation approaches or want to use source Qf 
origin (which ones were developed for adult education programs) or degree ' 
of familiarity. 

We decided to try to form a few general groups based on some aspect 
they had in common that set them apart from other models. 

First, the approaches that -gattema functioning ay stem were separated 
from those that focus on progipm reaults* The approaches that pattern a 
system were then divided into two categories — one dealing with patterning 
of programs; the other with patterning of evaluation. Each of these two 
categories was again subdivided. The program patterns were divided into 
those that emphasize evaluation as input into decision making and those that 
pattern parts of programs. The approaches that emphasize evaluative activities 
were divided into those that deal with kinds of data and those that describe 
processes of evaluation. Then the multitude of approaches that focus on 
program results were separated into those dealing with objectives and those 
dealing with a broader approach to outcomes and effects. 
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The following six groupings resulted: 

1* Evaluation as input into decision making. 

2. Evaluation of program parts. 

3. Evaluation — kinds of data; types of activities. 

4. Evaluation processes. 

5. Results-^attainment of objectives. 

6. Results — evaluation of outcomes and effects. 

The grouping provides one way of categorizing current approaches 
to evaluation. Table page 50, gives the location of various approaches within 
the 6 groups. Although several of the approaches could have been placed 
in more than one category, we grouped them according to their primary focus. 

The various approaches don't substitute for each other. Few are in 
conflict with each other. Many can be used in combinations of two or more. 
Even the approaches within a group aren't different ways of doing the same 
thing. For example, you could and perhaps should use all of the approaches 
in Group 1 over the period of a Y^^^^ The decision centered evaluation 
model (CIPF) provides a framework for dealing with the decisions necessary 
in programming over a span of time. Although it can be applied to a specific 
project of program phase in the same way that the developmental evaluation approaches 
are, it has additional value if applied to the total program operations. 
The differential evaluation model emphasizes judgments and conclusions 
more than decisions, but provides an approach to deciding what aspects 
of the program will be judged. It's useful in taking st'->ck of what's 
happened or what's happening at a given point in time. The diecrepancy evaluation 
model is useful in trouble shooting. If a program is ailing or if you don't 
really understand what's happening, the discrepancy approach is helpful. 
The priority decieione approach is a subpart of context evaluation. But 
because the ultimate worth of a program depends on the choices made at the 



very beginning, frameworks for evaluating the fundamental decisions of a 
program (which clientele, what content, how long, what approaches, etc.) 
are extremely important. Finally, most programs use texcs, films, or other 
materials and need ways of choosing among them. 

The second group of models can be used in conjunction with the first 
with no difficulty. The first group deals with qualities or stages of 
programming. The second deals with actual parts of that program. You 
could, for example, combine the actual component approach with the 
decision centered^ differential^ or discrepancy evaluation approach. 
The actual component approach would identify what elements within the program 
would be studied. The approach selected from Group 1 would establish the 
program sequence of examination and reexamination. The approach selected 
from Group 2 would indicate what element would be examined. 

The third group emphasizes parts much as the second does, but it*s 
talking about kinds of data that are used rather the . about the program 
part directly. Similarities and differences exist among the approaches 
in Group 3. For example, the countenance of evaluation and eyatem role 
model wero developed during the same period by men at the same university 
both applying system and role theory. The meane-^enda hierarchy provides 
a different way of looking at data f3?om the first two. It limits itself 
to the process and product categories of the decision centered evaluation 
models but distinguishes various categories of product data. 

The fourth group of approaches deal with processes. Three of them 
are basic processes that can be applied regardless of what it is about a 
program that^s being evaluated. The two traditional processes, appraisal 
that emphasizes human judgment and approaches that emphasize a high use of 
eyatematically collected data, aren^t altemativea to each other. The 
natural proceaa approach indicates that they can and should be used together. 
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The other three approaches deal with specific applications of process. The 
monitoring evaluation model stresses the continuous approach to evaluation 
using either appraisal or systematic data processing or both. The tm- 
provement evaluation ^uodel shows how appraisal can be used interactively 
to stimulate programmers and the tvansaotional evaluation emphasizes an 
approach that will help staff accept recommendations for chapge. 

The fifth group is the most highly developed. Much more attention has 
heen given to t\ii\ details related to evaluating results using attairmient 
of objectives as -che basic criteria of success than to any of the other forms 
of evaluation. Rather than emphasizing a few of the several developments, this 
section gives a fa.rly concentrated overview of what has happened related 
to objectives* 

The s'xth group again gives more space to the individual approaches. 
Most of the approaches to results that are broader than objecti'^^s try . 
to define and describe the types of outcome and effects that make 
program results. 

Groups 5 and 6 can be subsumed Kithin the approaches in Groups 
1, 2, and 3. They're ways of spelling out the results component in more 
detail when you're concentrating on taking a program system approach to 
evaluation, with results being one component in that system. On the other 
hand, if you're particularly concerned with results and trying to explain 
the degree of success in producing results, some of the frameworks from 
Groups 1, 2, and 3 will be useful in addition to those within Croups 5 ojid 
6 that try to use program processes or elements as explanatory variables. 
Although it's unwise to separate results from other program elements in that 
you lose much in understanding how a program produces or fails to produce resuJ.ts, 
sometimes cutside pressures for results data and lack of resources force you 



to limit exploration only to examining the results of a program* Approaches 
that emphasize results have been left as separate categories, but have been 
included as part of program evaluation rather than defining it as a kind of 
evaluation important in its own right, but different from program evaluation. 

At this point, unless you're already familiar with some of the approaches, 
the details in the last few paragraphs that have discussed the various groups 
of approaches may have little meaning. Don*t worry about it. The intent of 
the paragi^aphs is to underscore the idea that the various approaches to evalua-- 
tion seldom deal with the same need. Thp search shouldn't be for finding 
a way, but for building a repetoire of ways of evaluating. 

In this section and in the summaries in Section III, the approaches aren't 
specifically discussed in terms of evaluating programs for the disadvantaged. 
Such programs range extensively in terms of their nature and their needs. The 
range is as great as in the whole of adult education. Therefore, the applica- 
tion of the approaches and the potential for using them isn't unique in this 
particular kind of programming. 

The Work That Lies Ahead 

Although you'll find this monograph very useful, it may not meet all 
of your needs. Its value is limited both by the way the monograph is 
structured and bv the present state of the theory, art, and practice of 
evaluation. 

iiimitations within the monograph include: 

1. The nature of the classification system. It's tentative and general. 
It may be misleading. 

2* Omission and distortion that comes through summarization. 3ome of 
the original sources are books, others are brief papers. In some 
instances, we've heard the author present the approach; in others 
we've relied only on written source materials. It's difficult to 
summarize and make the original ideas clear. 
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3. The lack of parallelism. . Even though a standard format is used 
in presenting the approaches, it*s very difficult to compare 
them. They aren*t at the same level or stage of development. There *s 
no consistency in overlap and duplication. It*s there, but not 
sufficiently .to make a clear pattern. 

The lack of clear procedural directions. In summarizing the approaches, 
priority has gone to the parts that outline the "what" and "why" 
of evaluation rather than those that deal with the "how" of carrying 
out procedure. Once you have a clear sense of direction and specifica- 
tion, it*s fairly easy to generate a process that will aim in the 
direction you want to go. Some of the sources give procedural de- 
tails. Others don*t. 

5, The lack of case examples. In some cases, the models haven't 
been completely applied yet. In others, it requires too many 
pages to do justice to a description of the application. In some 
instances, a particular framework can be applied in several ways. 
Just prese* :ing one way restricts you too much. Unfortunately 
most evaluation reports are designed to give information about 
programs, not to describe the evaluation. At best they stop with 
conclusions and recommendations. Few give the full story as to 
purpose, rationale* overall approach (procedure usually deals only 
with how data were handled), and what happened because of the 
evaluation. To understand the effectiveness of a particular 
evaluation approach, you have to know what occurred as a result of it. 

Various limitations in terms of the present state of theory, art, and 

practice in program evaluation have been indicated in the earlier discussions. 

A good deal more needs to be done in conceptualizing modeling and developing 

and testing guidelines and procedures. For example: 

1. Now that we've realized that accurate information is an input into 
evaluation but isn*t the whole of evaluation, much more attention 
must be given to translating the mental and social interactive 
activities of evaluation into procedures or guidelines. How do you 
assign value? Control trade-offs? Exactly how do you go about 
assigning meaning to data and forming conclusions? How do you 
evaluate the differences between alternatives and finally reach 
a decision? How do you develop a logical argument to support a 
judgment that's powerful enough to convince an antagonist? These 
answers aren't easily available in procedural steps and yet these 
are some of the activities that constitute the heart of evaluation. 
We need frameworks for making the mental activities of evaluation 
more systematic and accurate. Some of the procedures must be program 
specific. They need to be developed by a team representing all those 
who have interest in the evaluation — participant, programmer, 
administrator, legislator, social policy maker, local influential-*^ 
and designed to fit within a certain programming situation. Pro- 
fessional groups must take a more prominent role in developing 
standards and setting realistic input-^ut^ut ratios. 
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2. We need guides and procedures for helping the professional improve 
his ongoing, everyday evaluation and decision making. How can he 
sharpen his powers of accurate observation and interpretation of what 
he sees and hears occurring? How can he detecv biases and the 
possible effects of those biases? 

?. We need cost-benefit studies of the contribution of "hard" data—that 
which is gathered using research type techniques — to accurate 
evaluation. Are the judgments and decisions made when extensive 
amounts of "hard" data are available enough better than those made 
without such data to justify the costs involved? 

In the last five years we've seen a great deal of progress in how 
we examine the results of instruction, in particular, how we 
determine if* specific changes in knowledge have occurred. Similar 
developments are needed in terms of examining larger results — attain- 
ing program and institutional goals, accomplishing purpose and mission, 
meeting needs, solving problems. In adult education, it*s the larger 
ends that count. Changes in knowledge are of value only if they lead 
to the overall development of the adult and his ability to enjoy life 
and manage his environment. 

5. As we accept program evaluation as a generic term with many activities 
and many aspects of program on which it should focus, as we accept 
it as a toolbox full of tools rather than one tool, more work needs 
to be done on designing the various tools in that toolbox. First more 
work must be done on identi^ing the kinds of tools that are rjost needed 
and building some kind of a classification system that*s useful. The 
classification system will help us organize the developments so we 
understand their different uses and purposes. It will help some of 
the confusion that occurs when we discuss the ideas of Tyler, Stuffle- 
beam. Stake, and Scriven as though they're all dealing with the 
same thing. Then very serious work needs to be done on the activities 
that come in the various categories. For example, tools dealing with 
the attainment of instructional objectives are shaping up nicely. On 
the other hand.. we*re just beginning to rough out designs for evaluating 
needs and determining program priorities;. This is a task where the 
programmer must take the lead rather than passing the ball to the 
professor. 

Apparently we've moved from the era when evaluation could be compared 
with the first horseless carriage to the stage where it goes back to a 
variety of drawing boards for the kind of work that will produce 1975 models* 
We now know that the horseless carriage we've been using for the past 30 
years can be further developed. And we have some leads about the kinds of 



improvements that should be made* 
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But improvement of the theory and the science of evaluation won't be 
of much value unless improvements are made in the context in which it*s expected 
to operate. Improved theory and procedures of evaluation won't settle con- 
flicts between administrative levels in terms of what kind of evaluation is 
desired. It won't solve pr»oblems caused by faulty assumptions about what It's 
possible for programs to do. Some of the needs related to evaluation have to 
be dealt with in broader contexts using research findings and good sound common 
S3nse. 

Although considerable work still must be done, the progress that has been 
made in the past few yeax?s is extremely heartening. You'll find a good deal 
of help in the approaches summarized later in the monograph; ideas and help 
that weren*t available to you 10 years ago. The wheels , of progress are 
indeed turnings 



Table 2 

GROUPING OF APPROACHES TO PROGRAM EVALUATION 



Group 1: Evaluation As Input Into Decision Making 

1.1 Decision Centered Evaluation (CIPP) [Stufflebeam; Phi Delta Kappa] 

1.2 Differential Evaluation [Tripodi, Fellin, 6 Epstein] 

1.3 Discrepancy Evaluation [Provus] 
lA Priority Decisions [Boyle] 

1.5 Developmental Evaluation 

IPI Formative Evaluation [Lindvall 6 Cox] 

NewStart Evaluation System [Lamrock, Smith, 6 Warren] 

1.6 Materials Evaluation 

Trade-Off and Comparative Cost Approach [Glass] 
Weighted Criteria Approach [Crane 6 Abt] 
1*7 Participant Reaction Approaches 



Group 2: Evaluation of Program Parts 

2.1 Execution-Impact Approach [Freeman 6 Sherwood] 

2.2 Actual Component Approach [Knox, Mezirow, 6 Darkenwald, Jr.] 

2.3 Managerial Systems 

Program Evaluation and Review Technique (PERT) 
Organization As a Total System [Young] 
Macro System Model [Alkin] 

System Approach to Goal Setting [Van Gigch 6 Hill] 
Program Contact System [Duft] 
Management Infomnation Systems (SEMIS) 
Evaluating Decision Making [Hesseling] 
2 A Socio-Organizational Systems 

Social Systems Models [Loomis] 

Organizational Models [Etizioni; Schulberg 6 Baker] 
Motivational Model [Lewis] 

Group 3: Evaluation— Kinds of Data; Types of Activities 

3.1 Countenance of Evaluation [Stake] 

3.2 System Role Model [Knox] 

3.3 Means-Ends Hierarchy [Bennett] 

3.4 When-To-Do-It-Yourself Continuum [Alexander] 



Group 4: Evaluation Processes 

4.1 Appraisal Model [Harris] 

4.2 Data Management [Phi Delta Kappa] 

4.3 Natural Process Approach [Steele] 

4.4 Monitoring Evaluation [Bruce] 
4*5 Improvement Evaluation [Kreitlow] 
4*6 Transactional Evaluation [Rippey] 
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Group 5: ResuUs^-Attainment of Objectives 

5.1 Tylerian Models 

The Original Tyler Model [Tyler] 
National Assessment Program 
Four-Question Approach [Gottman 6 Clasen] 
Adoption of the Tyler Model by Adult Education 

5.2 Instructional Evaluation Approaches 

Goal-Referenced Instruction [Popham 6 Baker] 

Evaluation As Facilitation of Learning [Bloom, Hastings, 6 Madaus] 

5.3 Program Objectives Approaches 

Criteria of Success Approach [Suchman] 

0-A-R Model [Deniston et al, ] 

Criteria and Objectives [Matteson] 

Locally Directed Evaluation [Byram & Robertson] 

Ohio Model [Starr et al,] 

Program Planning Budgeting System (PPBS) 

System Variables A;>proaches 

NEA Model [Taba 6 Sawin] 

Cube Models [Hanunond; Armstrong et al.] 

Information Domains [Nelson] 
Types of Evaluation [Wholey et al,] 
5.^4 Objectives: Categorizations and Criteria 

Objectives As Systems and Parts of Systems 
Level and Time Sequence of Objectives 
Kinds of Objectives Operating Within a Program 
Characteristics of Objectives 

Group 6: Results: Evaluation of Outcomes and Effects 

6.1 Goal Free Evaluation [Scriven] 

6.2 Zones of Results [Lamrock, Smith, S Warren] 

6.3 Multiple Change Approach [Hayes, Jr.] 

6.i+ Multiple Dimensions of Program Effectiveness [Steele] 

6.5 Effectiveness of Methods [Wilson 6 Gallup] 

6.6 Impact Evaluation [Borus 6 Tash] 

6.7 Public Policy [Berlak] 

6.8 Institutional Evaluation [Forehand] 

6.9 Social Indicators [Paulson] 

6.10 Research Models [Weiss; Longest; Cain £ Hollistev] 

6.11 Efficiency Examinations 

6 . 12 Accountability 




Section II 

MATCHING EVALUATION APPROACHES TO NEEDS 

As you enter your exploration of the evaluation approaches siimraarized 
in Section III, you face questions like: "How do I use evaluation approaches?" 
"How do I choose which one to study first?" 

Using Evaluation Approaches 
First, let^s talk generally about use. Familiarity with various 
approaches to program evaluation can be useful to you in at least four ways: 

1. Expanding your ideas about prograraming and about program evaluation 
so that you're better able to make evaluation an effective management 
tool • 

2. Helping you track down sources and references that consultants and 
others use when talking about evaluation. 

3. Providing approaches you can adapt to your own situation. 

^. Providing support and ideas for building your own approaches. 

Expanding Your Ideas About 
Prograrnming and Program Evaluation 

The help that current approaches to evaluation give you in understanding 
the dynamics and complexity of programming may in the long run be much more 
useful to you than the help that they give you in actually doing evaluation. 
Most of the approaches try to identic key components and the relationships 
among those components. The various patterns help you get a grip on the 
important ingredients and processes within programming. They help you 
identic the crucial aspects of successful programming. They help you under- 
stand the nature and 3?ole of criteria and, if the criteria are sound guides 
to success, result in more successful programs. A sound understanding of 
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the bases of program evaluation can lead to program improvement, even 
though evaluation isn't done in a conscious or systematic manner. 

Locating Sources 

Each field has its ovm jargon and names to be dropped. Have you ever 
found yourself frustrated when you attend conferences, deal with evaluation 
consultants, or talk with some of your colleagues who have a keen interest 
in conceptualizing evaluation because you aren't aware of who the names and 
what the ideas are? If this occurs frequently, you may want to use this 
monograph as a briefing device. An author index and a title index have been 
included at the end to help you better use it for such purposes. In all 
instances where the author has assigned a title, that title has been used. 
However, in some instances a label had to be arbitrarily assigned in the 
developmenc of the monograph. Appendix A on page 220 provides a brief 
paragraph description of each approach organized alphabetically according 
to the author. 

Providing You with Approaches You Can 
Adapt to Your Own Situation 

In some instances, you'll find that an approach pretty well fits a need 
and you'll try to use it. In the case of the more comprehensive models, the 
extent to which you deal with all the parts of a particular pattern depends 
on the depth of the evaluation and the extent of your resources. 

For example, at the everyday, "natural" level, you probably can make 
some observations and judgments about all of the parts of a particular pattern 
and the way in which they fit together. You may be able to operationalize 
a complete pattern without undue difficulty once the pattern is firmly in 
mind. However, if you're going to use a great deal of system-^-rigorous 
definition of criteria and carefully controlled gathering of data — you may 
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not have resources to adequately cover all of the approaches* components 
with the same depth of attention. This is particularly true if both the 
program and the evaluation approach is fairly complex. Some models such as 
the eountenanae of evaluation^ the system role models and the meana^ends 
hierarchy weren't meant to be used in their entirety in normal systematic 
evaluation situations. They were designed to help you understand the types 
of data to select from in developing a meaningful evaluation. 

In most instances you will select certain parts of a pattern for systematic 
examination. There's a growing push toward selective evaluation. For example, 
R. E. Brack of the University of Saskatchewan suggests that you take an 
eclectic approach — ^first identify the questions about the program that need 
to be answered and then select the parts of a particular model that can 
help deliver those answers without trying to systematically operationalize 
the complete model. In this situation, however, an understanding of the 
total pattern helps you keep the component that's receiving major attention 
within a total perspective of programming relationships. 

Don't try to operationalize an approach by just using the material in 
this monograph. Go to the original printed source. If possible, after 
you've studied it thoroughly, consult the author of the model. The summaries 
presented here are guides to approaches rather than guides for doing evaluation* 

How helpful the approaches will be when actually carried out depends on 
(l) the nature of your need, (2) how you conceptualize your program operations, 
(3) what judgments about program are most important in your situation, (if) what 
help in making program decisions or what other value can be gotten from a 
particular approach, and (5) how able you (and your evaluation consultant, if 
you use one) are to work from a working description rather than a precise 
operational blueprint. 
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Most of the time you will have to adapt the approach to your own situation 
rather than applying it "straight." However, be careful the adaptations don't 
distort the original model so it loses its value or becomes inappropriate. 

Providing You with Support and Ideas 
for Building Your Own Approaches 

In many instances, rather than extensively adapting a particular approach, 
you might be better off to construct your own, borrowing the parts of other 
approaches that are most useful and building patterois and processes that are 
appropriate to your needs • 

The approach you use should stand up to tests of internal and external 
validity, reliability, relevance, scope, creditability , pervasiveness, and 
timeliness. Most of these qualities are situation specific. A model may 
have them in one application but not in another. If it*s to be used by others? 
remember that you must be able to conununicate it clearly to them. 

Most important, you must be able to make it work so it will give reliable 
and believable results with a minimum of strain on the people involved. 

Choosing Evaluation Approaches 

Very few readers will or should try to read Section m from beginning 
to end at one sitting. Clf you should want to examine all of the approaches, 
you* 11 probably function better by working through a group at a time, 
starting with a particular need or interest in evaluation and going 
directly to the approach or approaches that best relate to that need.) 
Caution: Don*t be channeled completely by one need or view of what evaluation 
is. Remember that evaluation is a generic phenomenon and that it includes 
several approaches . Don aearah for the one Way to do evaluation. Do 
eearah for the range of approaohee -that Dill beet addreee your varied needs 
in program evaluation. 
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Start your evaluation approaches by outlining the kinds of needs that you 
and your unit have that evaluation "can help with. It may be helpful for 
you to take a dual track. On one sheet, list the Hnde of evaluation frameDorka 
that you need if your kit of evaluative tools is to be complete. On another 
sheet, list the epeoifio probleme you and your unit face where evaluation can 
be of help. 

Types of Evaluation Frameworks 

A program unit needs a group of frameworks to be applied frequently and 
consistently — for example, a framework for doing the kinds of evaluation 
that are essential throughout the program process and for dealing with every- 
day management questions; a framework that views the program through a 
participant's eyes; a framework that monitors the effectiveness of the various 
components of a program. These types of frameworks don't substitute for 
each other. They need to be tied together through a conunon philosophy of 
what evaluation is and an understanding of the general process of evaluation 
and subprocesses such as appraisal and data management # In addition, the 
program unit may also need some frameworks for dealing with more specific tasks 
like: choosing visual aids, texts, and other support materials; establishing 
priorities for programs; introducing change into a system. Table 3 provides 
examples of how various approaches summarized in the monograph relate to 
needs for general kinds of evaluative frameworks. 

Some frameworks are more important to one program than to another or 
are more important at a particular point in time. For example, programs for 
disadvantaged adults more than other programs may need frameworks for seeing 
how program participants evaluate the program or may need to use such 
frameworks extensively during the first two years of operation but give 
less attention to them in later years. Pvx>grams that develop all of their 
own materials won't need any frameworks for evaluating materials to be purchased. 



Table 3 

COMMONLY NEEDED EVALUATION FRAMEWORKS 



Most programmers and program u.'its need more than one evaluation 
framework in their tool kit. Different approaches accomplish different things. 



Types of evaluation frameworks Examples of approaches Included On 

needed by program units in this monograph page: 

Frmeworke that help mcke Decision Centered Evaluation (CIPP) 78 

major deaieions and guides Differential Evaluation 81 

overall program management Criteria of Guccess Approach i7i 

0-A-R Model 173 
Program Planning Budgeting System 178 
(PPBS) 

Frameworks for examining the Multiple Change Approach 198 

impact and larger reaulte of Multiple Dimensions of Program 200 

programs Effectiveness 

Impact Evaluation 203 
Social Indicators ' 210 
Means-Ends Hierarchy . 129 

Public Policy 205 

Research Models 212 

Criteria of Success Approach 171 

Effectiveness of Methods 202 

Zones of Results 195 

Efficiency Examinations 215 

Accountability 217 

Ohio Model 177 

Criteria and Objectives 174 

Types of Evaluation IBI 

Frameworks that guide the Actual Component Approach 104 

organization and use of Program Evaluation and Review 107 

program components Technique (PERT) 

Macro System Model no 

Organization As a Total System 109 

Countenance of EvaJ.uatlon 120 

System Role Model 12** 

Social Systems Models 116 

Organizational Models 117 

Program Contact >ystem 112 

Management Information Systems 11^ 
(SEMIS) 

System Variables Approaches 179 
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Types of evaluation frameworks Examples of approaches included On 
needed by program units in tnis monograph . Page: 



Frcmeworks for viewing the 
program through the eyes of 
the partieipant 



Participant Reaction Approaches 98 

Locally Directed Evaluation 176 

Motivational Model 118 

Transactional Evaluation 150 

Institutional Evaluation 207 



Frameworks that guide Appraisal Model 135 

evalx4ative proaesses Data Management 138 

Natural Process Approach 143 

Monitoring Evaluation 146 

Locally Directed Evaluation 176 

When-To-Do-It-Yourself Continuum 131 

Frameworks for examining Tylerian Models 154 

the results of inetruotion Goal -Referenced Instruction 164 

Evaluation As Facilitation of 168 
Learning 

Goal Free Evaluation 192 

Institutional Evaluation 207 

Objectives: Categorizations and 184 
Criteria 

Adoption of the Tyler Model by 157 
Adult Education 

Frartieworke that serve specific Discrepancy Evaluation 84 

purposes Priority Decisions 86 

Developmental Evaluation 89 

Materials Evaluation 95 
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However, adult basic education which draws on many commercially prepared 
materials may find such a framework essential to effective operation. Some 
programs may need frameworks for examining instructional results more than 
they need frameworks for examining larger results. On the other hand, other 
programs for disadvantaged adults need to concentrate cn exploring major 
import and value, and if resources don*t stretch to cover both, give less 
attention to the specific results of instruction. 

Each program unit has to decide which frcimeworks are most relevant and 
important to the operation of the particular unit, and as a result, which 
frameworks should be extensively operationalized and which should be drawn 
on periodically or at time of greatest need. Even though a program unit 
restricts itself to two or three general frameworks, if this restriction 
is made after a full survey of the kinds of program evaluation that could 
and perhaps should be operating, there is a much better understanding of 
what the particular framework can (and can*t) be expected to do. 

Use of an Approach to Deal with a Specific Problem 

A second approach to choosing evaluation approaches is that of using an 
approach to deal with a specific problem, ^or example, the administrator 
recognizes that changes are needed in a program that staff are apt to 
resist. In such an instance, an approach like tranaaotiondl evaluation 
might be particularly helpful. Or, if staff feel uneasy about a particular 
progr^im phase but are unable to really diagnose the problem, dCearepcxnay 
evaluation could be helpful. Or, if staff need a challenge plus a pat on the 
back to give them further impetus, the inprovement evaluation approach would 
be useful. In most instances, the problem probably will re?5Ult in one of the 
major frameworks accepted by the program unit being applied, but in other 



instances like the ones given above, an additiondl evaluative approach not 
comroonly part of the tool kit may be selected. 

Table H includes exaiaples of problem situations that sometimes arise 
and indicates the approaches within the monograph that might be useful in 
dealing with those situations. They have been loosely grouped into three 
types: problems in programming, problems in program management, and problems 
in evaluation. 



Table 4 

MATCHING EVALUATION APPROACHES TO PROBLEMS AND NEEDS 



If you're having problems in PR0(3tAMGNG . . . 



• • « < 

wiT:n: 



Setting pHoHtiee 



Look for ideas In: 



Priority Decisions 
Decision Centered Evaluation 
(CIPP) [context] 



On page:- 



86 
78 



Chooeing camng program 
poaaibi litiee 



Decision Centered Evaluation 

(CIPP) [context] 
Priority Decisions 



78 
86 



Determining the kind of Means-Ends Hierarchy 129 

resultB you* re aiming for System Role Model 12U 

OB you deeign the program Multiple Change Approach 198 

Multiple Dimensions of Program 200 

Effectiveness 

Impact Evaluation 203 

Research Models 212 

Social Indicators 210 

Goal-Referenced Instruction 164 

Evaluation As Facilitation of 168 
Learning 

Object J.ves: Categorizations 184 
and Criteria 



j developing obgectivee 

1 



i 

i 

; Determining the right 

level of objeotivee 



System Approach to Goal Setting 111 

Decision Centered Evaluation 78 

(CIPP) [context] 

Priority Decisions 86 

Ob j ectives t Categorizations 184 

and Criteria 

Tylerlan Models 154 

Objectives: Categorizations 184 

and Criteria 

Criteria of Success Approach 171 



ll ^This table is divided into three main sections — ^problems in programming$ 

{ problems in program management, and problems in doing evaluation. Groupings 

I are general and not mutually exclusive* Models suggested as sources of ideas 

I for a particular task are examples and not the only 'relevant ones* 

I 

^ -61 
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If you •re having problems in TEOGRAMMIHG . . . (cont.) 



With: 



Look for ideas in: 



On page: 



Selecting content and foam 



Decision Centered Evaluation 

CCIPP) [context] 
Participant Reaction Approaches 
Program Contact System 



78 

98 
112 



Identifying what participante 
see 08 appropriate progxxxm 
activities 



Participant Reaction Approaches 



98 



Examining factors affecting 
participation and learning 



Decision Centered Evaluation 

(CIPP) [context] 
Social Systems Models 



78 
116 



Developing a project or 
program plan 



IPI Formative Evaluation 
NewStart Evaltiation System 
Program Evaluation and Review 
Technique (PEPT) 



89 
91 
107 



Developing a pilot project 



Discrepancy Evaluation 
NewStart Evaluation System 
Social Systems Models 
Tylerian Models 



84 
91 
116 
154 



Determining how you can use 
scarce resources most 
effectively in the program 



Decision Centered Evaluation 

(CIPP) [input] 
Differential Evaluation 

[efficiency] 



78 
81 



Making sure that all of the 
elements of the program 
are appropriate 



Appraisal Model 



135 



Understanding how various 
aspects of the program fit 
together 



Actual Comno.ient Approach 
Program Evaluation and Review 
Technique (PERT) 



104 
107 



i 
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If you ^ re having problems in PROGRAMHTltG . . . (cont.) 
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With: 



Look for ideas in: 



On page: 



Working effectively with othere 
in the pj^-ogrcm 



Motivai'ional Model 
Transactional Evaluation 
Organizational Models 



118 
150 
117 



Choosing among textbooke^ 
audio-viaual paokageB, eta. 



Trade-Off and Comparative 

Cost Approach 
Weighted Criteria Approach 



95 
96 



Prepcan^ng teaching plana 



Goal-Referenced Instruction 
Evaluation As a Facilitation 

of Learning 
Objectives : Categorizations 

and Criteria 



164 
168 

184 



Determining whether a pro-- , 
gram ie progreaaing toward 
ita goala 



Monitoring Evaluation 



146 



Finding out why a program 
ian^t producing aa much aa 
you expect it to 



Discrepancy Evaluation 



84 



Finding out how well your 
firat programming efforta 
are achieving 



Differential Evaluation 
[initiation and contact! 

Monitoring Evaluation 



81 
146 



Clarifying what kind of 
reaulta to expect from a 
new program 



Goal Free Evaluation 
Research Models [search 
evaluation] 



192 
212 



Underatanding how pwgram 
componenta interact to 
produce reaulta 



Actual Component Model 
Macro System Model 
Countenance of Evaluation 
System Role Model 
Program Evaluation and Review 

Techniiuo (PERT) 
System Vjtriables Approaches 



104 
110 
120 
124 
107 

179 



I 

! 

I 

If you*re having problems in PROGRAMtlNG . . . (cont.) 



With: Look for ideas In: On page; 



Testing a particular inetrua" Tylerian Models 15«f 

tionat sequence to eee what it 'a Goal-Referenced Instruction 16«f 

achieving Evaluation As a Facilitation of 168 

Learning 

Selecting the moat effective Effectiveness of Methods 202 
method for a particular taek 

Determining the effect of a Impact Evaluation 203 

program on a conrtunity Social Indicators 210 

Trade-Off and Comparative Cost 95 
Approach 

Identifying whether you^re Execution-Impact Approach 102 

reaching the right clientele Participant Reaction Approaches 98 

Adoption of the Tyler Model 157 
by Adult Education 

• 

Revising a pilot program and Execution-Impact Approach 102 

developing a guide for others Discrepancy Evaluation 8«f 

c to follow Macro System Model 110 

' NewStart Evaluation System 91 

System Role Model 12«f 

Getting other people '« ideas Appraisal Model 135 

on how your program can be Improvement Evaluation 1^8 

improved Participant Reaction Approaches 98 

Improving on-the-spot judgments Natural Process Evaluation W3 
as a program progresses 

Examining the results of Decision Centered Evaluation 78 
programs (CIPP) [context] 

Trade-Off and Comparative Cost 95 
Approach 

\ Execution-Impact Approach 102 

\ CountencUice of Evaluation 120 

i System Role Model 124 

i Means«-Ends Hierarchy 129 

I Tylerian Models 15tf 

f Instructional Evaluation 207 
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If you're having problems in PROGBAMMISG . . . (cont.) 



With; Look for ideas in: On page: 



Examining the reaulte of Program Objectives Approaches 171 

program (cont.) Goal Free Evaluation 192 

Zones of Results 195 

Multiple Change Approach 198 

Multiple Dimensions of Program 200 

Effectiveness 

Impact Evaluation 203 

Institutional Evaluation 207 

Social Indicators 210 

Research Models 212 

Efficiency Examinations 215 



Identifying key elemente Actual Component Approach 104 

contributing to the euooeee Macro System Modal 110 

or failure of a program Improvement Evaluation 148 

System Variables Approaches 179 

Countenance of Evaluation 120 

Means-Ends Hierarchy 129 



Preparing a program report 



Data Management [communication] 138 
Trade-Off and Comparative Cost ^ 95 
Approach 



Deoiding what to emphasize 
in a report prepared for 
adninietratore and influentiale 



Multiple Change Approach 198 

Multiple Dimensions of Program 200 

Effectiveness 

Impact Evaluation 203 



If you're having problems in PROGRAM MANAGEMENT • 



Guiding a program that involves 
eeveral etaff menibere with 
rolee dependent on prior work 
by othere 



Program Evaluation and Review 

Technique (PERT) 
Organizational Models 



107 
117 



Understanding the kind of 
decisions involved in 
programmtng 



Decision Centered Evaluation 

(CIPP) [input] 
News tart Evaluation System 
Evaluating Decision Making 



78 

91 
115 



letting the most out of 
t limited resources 

I 

;■) 
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Decision Centered Evaluation 

(CIPP) [input] 
IPI Foi»mative Evaluation 89 
Participant Reaction Approaches 98 
Trade-Off and Comparative Cost 95 

Approach 
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If you're having problems in PROGRAM MAMGEMENT . . . (cont.) 

With: Look for iddas In; On page: 



Deciding whether ideas for Priority Decisions 86 

program are good Decision Centered Evaluation 78 

(CIPP) [input] 

IPI Formative Evaluation 89 

NewStart Evaluation System 91 

Goal Free Evaluation 192 

Trade-Off and Comparative Cost 95 
Approach 

Supervising aotivitiee of Execution-Impact Approach 102 

several para professionals System Role Model 124 

Motivational Model 118 

Securing additional funding Data Management [Communication] 138 

Execution-Impact Approach 102 

Differential Evaluation 81 

[effectiveness, efficiency] 

Program Objectives Approaches 171 
Multiple Change Approach • 198 

Multiple Dimensions of Program 200 

Effectiveness 

Accountability 217 

Trade-Off and Comparative Cost 95 

Appa?oach 

Decision Centered Evaluation (CIPP) 78 

Developing long-range System Approach to Goal Setting 111 

goals and miesion Priority Decisions 86 

Multiple Change Approach 198 

Objectives:' Cafegcrizat ions and 184 
Criteria 

IPI Format i%e Evaluation 89 

Reviewing long-range IPI Formative Evaluation 89 

goals Multiple Change Approach 198 

Priority Decisions 86 

Goal Free Evaluation 192 

Improving teanwork within Social Systems Models 116 

your unit Motivational Model 118 

Organization As a Total System 109 

Transactional Evaluation ISO 



i 

( 
I 

I If you're having problems in PROGRAM MANAGEMENT . • • (cont.) 

I 

] With: Look for ideas in: 



Vndevatanding the atagea Differential Evaluation 81 

of programming Program Contact System 112 

Understanding relationship Decision Centered Evaluation 78 
of evaluation to deoiaion (CIPP) 

making NewStart Evaluation System 91 

Trade-Off and Comparative Cost 95 
Approach 
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On page: 



Defining program effiaienoy Efficiency Examinations 215 

Differential Evaluation 81 

Criteria of Success Approach 171 

Execution-Impact Approach 102 



Settling disputes when tuo 
or more methods or plans 
are advocated for doing 
the same thing 



Decision Centered Evaluation 

(CIPP) [input] 
Differential Evaluation 
Effectiveness of Hethods 



78 

81 
202 



Designing a management 
information system for 
your unit 



Management Information Systems ll'^ 
(SEMIS) 

System Role Model 12«f 

Data Management 138 



Designing acoountahility Accountability 217 

strategies. Decision Centered Evaluation 78 

(CIPP) 

Locally Directed Evaluation 176 

Types of Evaluation 181 

Management Information Systems 11^ 
(SEMIS) 

Data Management 138 



Developing an organization Motivational Model 118 

and climate that toill Organizational Models 117 

improve programming Social Systems Models 116 

Transactional Evaluation 150 



Identifying and weighing Priority Decisions 86 

groups and other pressures Social Syatems Models 116 

that must be considered 
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If you* re having problems in PROGBAM MANAGEMENT . . . (cont-) 



With: 



Look for Ideas In: 



On page; 



Getting new ideas adopted by 
the etaff 



Motivational Model 
Discrepancy Evaluation 
Transactional Evaluation 



118 
150 



Organizing eidpport veeourceB 
effectively 



Organizational Models 
Program Evaluation and Review 
Technique (PERT) 



117 
107 



Increasing operational 
efficiency 



Efficiency Examinations 215 

Differential Evaluation 81 

Program Evaluation and Review 107 

Technique (PERT) 

Program Planning Budgeting 178 

System (PPBS) 

Criteria of Success Approach 171 

Trade-Off and Comparativ2 Cost 95 

Approach 



Cooperating with content experts 



Program Contact System 
Motivational Model 
Transactional Evaluation 
Discrepancy Evaluation 



112 
118 
150 



Using a systems approadh to 
organizing progranming 



Organization As a Total System 109 

Actual Component Approach 10^ 

Macro System Model HO 

Countenance of Evaluation 120 

System Role Model 12^* 

Program Evaluation and Review 107 

Technique (PERT) 

System Variables Approaches 179 



Developing criteria to guide 
the work of your unit 



Criteria of Success Approach 171 

Natural Process Approach 1**3 

Differential Evaluation 81 

Discrepancy Evaluation 8** 

Multiple •)iroensions of Program 200 

Effectiveness 

Public Policy 205 

Institutional Evaluation 207 

Trade-Off and Coniparative Cost 35 

Approach 



ERLC 



69 



If you* re having problems in PROGRAM MANAGEMENT . • . (contj 



ERLC 



With: 



Look for ideas in: 



On page: 



Considering inputs proaeesj and 
output vetationehipa of 
vaHouB parts of your program 



O-A-R Model 

Actual Component Approach 
Organization As a Total 

System 
Macro System Model 
System Approach to Goal 

Setting 
Countenance of Evaluation 
System Role Model 



173 
lOU 
109 

110 
111 

120 
124 



Developing a mlticourae 
curricmla or a multiaativity 
program 



Institutional Evaluation 
Decision Centered Evaluation 
(CIPP) 

Trade-Off and Comparative 
Cost Approach 



207 
78 

95 



Increasing the aontributione 
of voVunteera 



Motivational Model 
Social Systems Models 



118 
116 



Improving everyday operat%ons 



Natural Process Approach 
Trade -Off and Comparative 
Cost Approach 



lif3 
95 



Determining whether prograrming 
activities are on target in 
term of the unites mission 



Monitoring Evaluation 



146 



Using information in making 
program deoisions 



Data Management 

Natural Process Approach 



138 
143 



Assigning resources to staff 
units 



Program Planning Budgeting 

System (PPBS) 
Decision Centered Evaluation 

(CIPP) 

Efficiency Examinations 
Accountability 

Multiple Dimensions of Program 

Effectiveness 
Program Objectives Approaches 
Trade-Off and Comparative Cost 

Approach 



178 

78 

215 
217 
200 

171 
95 
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If you're having problems in PROGRAM MANAGEMENT • • • (contJ 



With: Look for ideas in; On page; 



Managing the teaahing-^teaming Evaluation As a Facilitation of 168 
tranaaation Learning 

Zones of Results 195 

System Variables Approaches 179 

Objectives: Categorizations 184 

and Criteria 

Monitoring Evaluation 146 

Natural Process Approach 143 

Trade-Off and Comparative Cost 95 

Approach 

Developing a pereonnet evaluation Multiple Dimensions of Prograis 200 

eyetem Effectiveness 

Appraisal Model 135 

Natural Process Approach 143 

Weighted Criteria Approach 96 

Trade-Off and Comparative 95 
Cost Approach 

; Analyzing wedkneeBee and Discrepancy Evaluation 84 

; probleme in the operation Program Evaluation and Review 107 
of the unit Technique (PERT) 

Decision Centered Evaluation (CIPP) 78 

Appraisal Model 135 

Improvement Evaluation 148 

I Trade-Off and Comparative Cost 95 

Approach 

Working with an advieory Locally Directed Evaluation 176 
oonmittee Decision Centered Evaluation (CIPF). 78 

' ! System Approach to Goal Setting 111 

: Social Systems Models 116 

I Accountability 217 

j Data Management [communication! 138 

' Monitoring Evaluation 146 

Natural Process Approach 143 

Appraisal Model 135 

Improving use of time and Program Evaluation and Review 107 
other resouroes Technique (PERT) 

:i Decision Centered Evaluation (CIPP) 78 

Trade-Off and Comparative Cost 95' 

H Approach 
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If you* re having problems in PROGRAM MANAGEMENT . . . (cont.) 



With: Look for ideas in: On page: 



Developing budgete Program Planning Budgeting 178 

System (PPBS) 

Impact Evaluation 203 

Decision Centered Evaluation (CIPP) 78 

Differential Evaluation 81 

Guiding and controlling the Differential Evaluation 81 

total programing proaeee Organization As a Total System 109 

System Role Model 124 

Program Contact System 112 

Helping staff use evaluation Natural Process Approach 1H3 

oa a management tool Monitoring Evaluation 1^6 

Decision Centered Evaluation (CIPP) 78 

Differential Evaluation 81 

Discrepancy Evaluation 84 

Trade-Off and Comparative Cost 95 
Approach 

If you're having problems in EVALUATION . • • 

Improving staff attitude Natural Process Approach 143 

toward evaluation Discrepancy Evaluation 84 

Decision Centered Evaluation (CIPP) 78 

Motivational Model 118 

Transactional Evaluation 150 

Distinguishing between des" Countenance of Evaluation 120 
orihing and evaluating pro- 
gram results 

Determining results when Goal Free Evaluation 192 

objectives were poorly stated Multiple Dimensions of Program 200 
or changed during the program Effectiveness 

Developing criteria for Appraisal Model 135 

^judging the program Natural Process Approach 143 

Discrepancy Evaluation 84 

Differential Evaluation 81 

Criteria of Success Approach 171 

Execution-Impact Approach 102 

Multiple Dimensions of Program 200 
Effectiveness 
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If you're having problems in EVALUATION . • . (cont.) 



With: 



Look for ideas in: 



On page; 



Developing ariteria for 
judging the program (cont J 



Institutional Evaluation 
Efficiency Examinations 
Accountability 



207 
215 
217 



Dealing with several ariteria 
at one time 



Weighted Criteria Approach 
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Forming judgmenta about program Natural Process Approach iH3 

Appraisal Model 135 

Decision Centered Evaluation (CIPP) 78 

Trade-^Off and Comparative Cost 95 
Approach 



Involving lay people in 
evaliuxtion 



Locally Directed Evaluation 176 
Participant Reaction Approaches 98 
Discrepancy Evaluation 



Organizing a compreheneive 
program review^ accreditation 
team J etc* 



Appraisal Model 135 

Improvement Evaluation 1H8 

Actual Component Approach lOH 

System Role Model 12H 

Organization As a Total System 109 
Decision Centered Evaluation (CIPP) 78 

Multiple Dimensions of Program 200 

Effectiveness 

Goal Free Evaluation 192 



Preparing for an accreditation 
team, monitoring team, or 
compreheneive review 



Appraisal Model 133 

Execution-Impact Approach 102 

Multiple Dimensions of Program 200 

Effectiveness 

Natural Process Approach 1**3 

Data Management [communication] 138 
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Determining what parte of 
the program to include in 
your evaluation 



Decision Centered Evaluation (CIPP) 78 
Actual Component Approach 

Appraisal Model 135 

Improvement Evaluation 1**8 

Countenance of Evaluation 120 

System Role Model 124 

Means-Ends Hierarchy 129 

Natural Process Approach 1**3 

Multiple Dimensions of Program 200 
Effectiveness 
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If you're having problems in EVALUATION . . . (cont.) 
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With: 



Look for ideas in: 



On page; 



Determining what parte of the 
program to include in your 
evaluation (cont.) 



Multiple Change Approach 
System Variables Approaches 
0-A--R Model 



198 
179 
173 



Identifying appropriate data 
baaea cmd aouroea of data 



Selecting uechniques to carry 
out specific evaluation taaka 



Categorizing kinds of judgmenta 
that need to be made about a 
program 



Deciding whetf* to conault an 
evaluation expert 



DeterminiAg whether an extensive 
research effort is appropriate 



Handuing o good deal of tm- 
r^^lated data a/ui making sense 
Gu'-' of how it fits into 
yztte-'^ns 



Setting p&rformanoe 
htandardo 



D^^'-hermining the kind of 
data needed 



NewStart Evaluation System 
Information Domains 



Differential Evaluation 



Differential Evaluation 
Natural Process Approach 
Criteria of Success Approach 
Criteria and Objectives 
Execution-Impact Approach 



When-To-Do-It-Yourself Continuum 



Types of Evci.l-:ation 

Decision Cer/,ered Evaluation (CIPP) 



Means-Ends Hierarchy 
System Role Model 
Countenance of Evaluation 
Actual Component Approach 



Countenance of Evaluation 
Appraisal Model 
Natural Process Approach 
Discrepancy Evaluation 
Goal-Referenced Instruction 
Multiple Dimensions of Program 

Effectiveness 
Criteria of Success Approach 
Differential Evaluation 



Natural Process Approach 

Counte^'-ance of Evaluation 

Means -Ends Hierarchy 

Decision Centered Evaluation (CIPP) 

Research Models 



91 
181 



81 



81 

143 
171 
174 
102' 



131 



181 

.78 



129 
124 
120 
104 



120 
135 
143 
84 
164 
200 

171 
81 



143 
120 
129 
78 
212 
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If you're having problems in EVALUATION . . . (cont.) 



7U 



With: 



Look for ideas in; 



On page; 



Establiahing or evaluating 
acrrtpling prooedurea 



Data Management 
Research Models 



138 
212 



Sszabliahing or evaluating 
treatment procedures 



L'^ta, Management 
Research Models 



138 
212 



Estabtiehing or evaluating 
measurement procedures 



Data Management 
Research Models 



138 
212 



Examining assumptione related 
to data 



Data Managemenc 



138 



Identifying flails and limita-^ 
tions in data as a basis for 
arguing against an unfavorab'^-e 
evaluation 



Data Management 



138 



Identifying flaws and limita'- 
tions in data as a basis for 
assessing the probable 
accuracy of the data 



Data Management 



138 



Developing evaluation reports 



Data Management [communication] 138 

System Role Model 124 

Trade-Off and Comparative Cost 95 
Approach 



Getting evaluation findings 
used 



Data Management [communication] 138 

System Rol^i M'^del 124 

Transactional Evaluation 150 

Discrepancy Evaluation 84 



Evaluating evaluation 



Data Management 138 
Trade-Off and Comparative Cost 95 
Approach 



Section III 

SUMMARIES OF CONTEMPORARY PROGRAM EVALUATION APPROACHES 

Thi3 section provides a brief sunnnary of contemporary approaches to 
program evaluation. The approaches are grouped into the following Categories 
according to the primary focus of the approach: 

Group 1. Evaluation as Input Into decision making* 
Group 2. Evaluation of program pa js* 
Group 3. Evaluation—kinds of data; types of activities* 
Group 4. Evfiluation processes* 
Group 5. Results --attainment of objectives* 
Group 6. Results— evaluation of outcomes and affects* 
Groups 5 and 6 emphasize examining the results of programs • The ot\er 
four groups focus on results and other elements of the program system in such 
a way that the information can be used in developing and improving programs 
while they^re in progress. A very brief introduction is provided to each 
group of approaches* 

The following information is given for each approach included within 
a group: field of origin, the essence of the approach, general suggestion as 
to use, place to start when using it, description of the approach, and a 
printed source where a more complete presentation can be found. 

The summaries should be treated as a ^•reference to" rather than a 
"reference foi.*' They* 11 help you decide wnich ones you* 11 want to get 
and study in greater depth or which authprs you* 11 want to contact directly. 
The summaries don*t provide a complete reference on how to use th^. particular 
approach. 



GROUP 1 

Evaluation As Input Into Decision Making 



OVERVim: 

What kinds of decisions -do you have to make about programs? What 
kinds of decisions do you make while you're programming? This group of 
models emphasizes evaluation's role in decision making. The models: (1) pattern 
types of decisions that have to be made, (2) identic program sequences 
that should be evaluated, C3) outline evaluation questions that need to be 
answered, or (^) identify criteria that should be applied. 

EXAMPLES Included: 

. 0 The decision centered evaluation model identifies four types of 

evaluation that correspond to the four major kinds of prograSnTdecisions. 

• Differential evaluation identifies three types of criteria that 
need to be applied to each of three programming stages. 

Discrepancy evaluation outlines a process by which standards are 
set for program activities and results and gaps identified by 
comparing actual performancs with those standarxis. 

0 The priority decieione approach describes six categories of criteria 

which may be important in deciding among needs and program alternatives. 

0 The two examples of developmental evaluation describe the process* • 
used ip developing program prototypes that will be used many times 
in many situations. 

^ The two examples of models for mteriale evaluation provide guides 
for choosing among packaged curricula. 

0 AeseeBing participant reaotione provides a summary of efforts of 
this type. 

GENERAL ADVANTAGES: 

0 Models like these help evaluation work for you in improving prograirs. 

0 They provide ways of dealing with decisions. 

0 They channel evaluation's energy back into ongoing programming. 
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GENERAL CAUTIONS: 

0 The models provide examples of patterns. Yoiir view of programming 
may not coincide exactly. For example, you may want to add other 
program stages, major decision categories, or kinds of criteria. 

0 As much emphasis should be placed on using ds is placed on securing 
information. 
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1.1 DECISION CENTERED EVALUATION (CIPP) 

ORIGIN Metropolitian and other local school systems. 

ESSENCE Four tjrpes of evaluation— -cyontext, input, prooeee, and produot---' 

correspond to four major kinds of decisions — planning, etruc- 
turing, implementing, and recycling. 

USE Used throughout programming, but particularly useful in: 

# Viewing total program over a period of time. 

# Developing new programs. 

# Understanding the decision dynamics within programs* 

STARTING Set decision parameters. What decisions must be made, by 

POINT whom, at what time? What information is needed for judging 

decision alternatives? 

DESCRIPTION Educational evaluation is defined as the process of delinea- 

ting, obtaining, and providing information useful for 
judging decision alternatives. It can be helpful at all 
four stages of decision making — m/kxreness, design, choijce, 
and action. It facilitates decision making by establishing 
criteria for distinguishing among alternatives. 

The CIPP model identifies four major types of evaluation 
that correspond to the four types of decisions involved in 
programming, 

C = Context — serves planning decision. 

I = Jnput— -serves decisions about designs and resources. 

P = Process — serves decisions that control operations. 

P = Product — serves decisions about results and re- 
cycling. 

Context Evaluation. Provides the rationale for determin- 
ing objectives and setting priorities- It defines the relevant 
environment, describes the desired and actual conditions 
pertaining to that environment, and identifies unmet needs 
and unused opportunities. It considers such things as be- 
havior of students, curriculum, staff strengths and weaknesses, 
facilities, financing, and the community. It examines 
present and emerging value systems and provides means for 
setting priorities. It examines the sanount of change needed. 
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Input Evaluation. Provides information for determining 
how to use resources to meet program goals. It deals with 
relevance, practicability, costs, projected effectiveness, 
and superiority alternatives. It involves identifying and 
assessing: (1) relevant capabilities of the system, 
(2) strategies for achieving program goals, and (3) designs 
for implementing a selected strategy. Two or more ways 
of carrying out the same program are analyzed in terms of 
expected costs and benefits. Has this kind of plan worked 
in the past? What are its underlying assumptions and can 
they be met? What side effects may be produced? How do 
people react to the approach? Can the plan actually be 
carried out successfully? 

Process Evaluation. Provides feedback. It*s concerned 
with the extent of operational efficiency including effects 
on rest of the system and feasibility. It detects or predicts 
defects in the procedural design and its implementation, 
provides information for decisions that are part of implementa- 
tion, and maintains a record of the procedures followed. 

It must be continually alert in advance to the kinds of 
decisions the teacher will be making. It*s concerned with 
such things as interpersonal relationships among staff and 
students, communication channels, logistics, resources » 
time schedules as potential causes of failures. The record 
of procedures helps to later analyze why certain- results 
occurred. 



Product Evaluation. Measures and interprets attain- 
ments during and at the end of the program. It*s concerned 
with main effects, side effects, costs, and superiority. 
Product evaluation involves establishing criteria, taking 
measurements, comparing measurement results with absolute 
or relative standards, and making rational interpretations 
of the outcomes using context, input, and process information. 

Criteria may be either ooneequential or inetrumental. 
Consequential criteria pertain to the fundamental long- 
range conditions being sought. Instrumental criteria deal 
with accomplishments at an intermediate level that contribute 
to the ultimate attainment. 

The four types can be used independently or in combinations. 
They can be used to facilitate decisions as the program 
progresses Cforroative evaluation) or in retrospective analysis 
of the quality of decisions that were made and implemented 
(summative evaluation and accountability). 



Formal, structured, comprehensive evaluation is most 
apt to be needed when the decisions involve major changes 
in areas where decision makers have little experience and 
little information. 

Evaluation models should be grounded in decision making 
and change theory. Evaluation designs should satisfy 
criteria of scientific adequacy (internal and external 
validity, reliability, and objectivity), of practical 
use (relevance, importance, scope, creditability , timeliness, 
and pervc ^iveness), and of prudential worth (efficiency). 



Phi Delta Kappa National Study Committee on Evaluation. 
Educational Evaluation and Decision Making* Itasca, 
Illinois: F. E. Peacock Publishers, Inc., 1971. 

Stuff lebeam , Daniel L. "Evaluation As Enlightenment for 
Decision-Making." Address at working conference on 
assessment theory, the Commission of Assessment of 
Educational Outcomes, the Association for Supervision 
and Curriculiim Development, Sarasota, Florida, January, 
1968. 

Stuff lebeam, Daniel L. »'The Relevance of the CIPP Evaluation 
Model for Educational Accountability.** Journal of 
Reeearoh and Developmnt in Education^ V (February, 1971), 
19-25. 
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1.2 DIFFERENTIAL EVALUATION 
ORIGIN Social fields. 



ESSENCE Systematic accumulation of facts about program achievement 

in terms of effort, effectiveneae, and efficiency within 
any of the stages of development— -initioticTrt, contacts 
and implementation. 

USE A means by which administrators and program personnel can 

decide where evaluation investments wi'^l be apt to give the 
greatest payoff in program improvement. Particularly 
usefTil in programs that haven't found the right routine 
or that aren't accomplishing as much as they would like. 

STARTING Determine key people. Secure understanding and commitment. 

POINT Examine possible consequences. Decide on the kinds of 

evaluative questions that will provide the most valuable 

information. 



DESCRIPTION Evaluation is a management technique for the systematic 

feedback of information to be used to improve social programs. 
Programmers who are committed to improving societal conditions 
through programs must contend with pressures froia a variety 
f of sources. In many instances, they must make quick decisions 
in ambiguous situations. The right information at the right 
time can be helpful. As the main- consumer of evaluation 
studies, administrators should play a major role in planning 
evaluation. 

The process of evaluation involves getting systematic 
information within a context of social relationships. The - 
people involved may hold different idealogies and value 
systems and various degrees of vested interests. A crucial 
element in obtaining useful evaluations is thts extent to 
which key people can accommodate one another. Examining 
political realities is as important in evaluation as it is 
in planning. Planning for soc5.al program evaluation depends 
a great deal on specifying program requisites within the 
different stages of development. Differential evaluatdt/i 
is a process of asking different evaluation questions for 
each stage of program development and then choosing those 
evaluation techniques most appropriate to the evaluation 
ol jectives. 

The three categories that define the kinds of questions 
. e; 

Program Efforts. Amounts and kinds of program activities 
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necessary with a particular stage of development; amount of 
resources invested and hw^w those resources were applied. 
Evaluation is concerned with the type and quantity of program 
activities . 

Program Effectiveness. Extent to which goals have been 
achieved; analysis of both desirable and undesirable un- 
anticipated consequences. Evaluation is concerned with 
whether these goals have been attained as a result of program 
efforts. 

Program Efficiency. The relationship between efforts and 
effectiveness. It's the ratio of effectiveness to effort 
and is concerned with the t^elative costs of achieving results. 
Costs include manpower, time, money, and facilities. Evalua- 
tion is concerned with the relative costs of achieving out- 
comes • 

The three program development stages are: 

Program Initiation, stage where ideas are translated 
into a plan of action an-i necessary resources are secured. 

Program Contact, stage where contact is made with target 
clientele; includes delivery system and analysis of obstacles, 
and establishing relationships with other programs in the 
community. 

Program Implementation. Energy is applied toi^ard the goals 
of the program. Concern is with results. Criteria for follow- 
up activities and possible program termination are specified. 

In these three stages: 

1. Longevity and complexity aren't indicative of a 
stage of prograia development. 

2. Different programs ^ithin the same agency may be in 
different stages of ^ development . 

3. Progression through the stages isn't always distinct 
and linear. 

U. Some programs may be concerned entirely with contact. 

The categories and stages produce questions like the following: 

Program Initiative Stage. To what extent are efforts made 
to locate anticipated target populations (effort)? To what 
extent does the program meet a comnunity need (effectiveness)? 
To what extent do staff members have competing objectives 
that are at cross purposes with one another (efficiency)? 
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Program Contact Stage. What amounts of time and energy 
are devoted to finding resources that could increase the 
number of program contacts (effort)? To what extent is the 
intended target population represented in those who are 
designated as program beneficiaries (effectiveness)? What 
are the relative costs of using different means for contacting 
clients (efficiency)? 

Program Implementation Stage. How much effort is devoted 
to the specification of criteria for program tertdnation and 
necessary follow-up activities (effort)? What^s the rela- 
tive effectiveness of the program compared with other programs 
that have similar objectives (effectiveness)? What^s the 
relation of costs of program efforts to the benefits achieved 
(efficiency)? 

The questions determine the kind of information-gathering 
technique to use. The most COTmion information-gathering 
techniques can be categorized as follows: 

Monitoring Techniques. Procedures that are used for 
direct review of program operations — accountability audit, 
time and motion studies, administrative audit. 

Social Research Techniques. Procedures used for 
developing, modifying, and expanding knowledge about the 
program — experiment, survey, case study. 

Cost Analytic Techniques. Procedures used to appraise 
the relative value of prograa benefits in relation to program 
costs — cost accounting, cost-benefit analy&is» cost-outcome 
analyses, and operations research. 

Tripodi, Tony, Phillip Fellin, and Irwin Epstein, ioaial 
Program Evaluation. Itasca, Illinois: V. E. Peacock 
Publishers, Inc., 1971. 
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1.3 DISCREPANCY EVALUATION 



Pittsburgh public school system and other education projects. 

Program personnel set etandca^ for activities at each 
programming scage — design^ inetdllatiortj proceeB, and 
product — and for the results expected. Actual performance 
is compared with the standards and diecrepcmeiee or 
areas for improvement identified* 

Particularily helpful in: 

# Checking out pilot or other new programs. 

0 Diagnosing programs that aren^t functioning well. 

0 Improving program efficiency. 

0 Helping new personnel better understand program develop- 
ment . 

Determine the key stages or points where standards should be 
applied. Devise the means by which the standards will be set. 

The following rationale underlies the approach to setting 
standards. Self -evaluation is a powerful device t perhaps the 
essential mechanism in changing individual and group behavior. 
A rational person will change his standards if they don't serve 
a value or are counterproductive to a higher value. An 
autonomous person must want to change hj.s standards before 
he will change his behavior. 

Standards can be changed through the analysis and synthesis 
of phenomena, insight, and the reorganization of a perceptual 
field, such as through the problem-solving process. It's 
possib"* . to get a person or group to change standards by 
showing discrepancies between performance and the assumptions 
and values that gave rise to the standard. 

A r.on-programraer (evaluation staff member) plays an important 
role in catalyzing the process by raising questions, supplying 
information that the programmer doesn't have time to collect 
and examine, and helping the programmer clarify values and 
biases • 

A program is a dynamic phenomenon that's made up of 
stages--500iflfn, installoHon^ prooess , and product. Each 
stage involves inputs j proosBSes t. and outputs^ At each 
stage, evaluation includes: U) defining program standards. 
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(2) determining whether a discrepancy exists between some 
aspect of program performance and the standards governing 
that aspect of the program, and (3) using discrepancy informa- 
tion either to change performance or to change program 
standards* 

Vfhen the diecrepanaiee have been identified, the next 
sequence involves determining: (l) why there's a discre- 
pancy, (2) what corrective actions are possible, and (3) what 
corrective action is best. Three elements are necessary in 
this process: (l) criteria for identifying relevant informa- 
tion and knowing how to interpret it based on some standard 
for the point of discrepancy to be investigated, (2) new 
information about actual performance and practice, and (3) a 
decision based on comparison of information and standard. 

At each stage the program unit has four alternatives 
for further program activities: (l) go on to the next stage, 
(2) recycle the stage after there has been a change in the 
program's standards or operations, (3) recycle to the first 
stage, or (4) terminate the project. The evaluation staff 
helps identi^ gaps or problems, but the program staff must 
decide what to do about them. 



SOURCE Provus, Malcolm. Dieorepanay Evaluation. Berkeley, 

California ; McCut chan Publishing , Co . , 1971 . 
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1.4 PRIORITY DECISIONS 



ORIGIN 
ESSENCE 



Cooperative Extension. 



This approach groups criteria for decision making about 
program priorities into six broad categories: eociety'^ 
cowmnityj clientele, political, organizational, reeoi4Tce0s 
and personal. 



USE 



STARTING 
POINT 



Although designed to be used in evaluating needs and 
alternative ideas for programs, it can also serve as a 
framework for examining the value and importance of 
completed programs . 



Identify which of the categories are most important as 
criteria sources for the particular program. 



DESCRIPTION Program development requires decisions. These decisions 

are often diificult. Programmers usually are faced with too 
many probleus to work on, too much content to teach, too 
many clientele groups to reach with the time and resources 
available. Therefore, priorities must be set* 

Criteria are needed for comparirg and selecting from 
among program alternatives. The specific criteria ro be 
used must evolve from the key influences affecting the 
particular program and programming situation. However, 
there are at least six general criteria areas that may 
need to be considered in establishing priorities. They 
include: 

SOCiety-Conmunity. r.e large society, with ICS institu- 
tions, governmental units, pressure groups, trends, goals^ 
and vaJ.ues, provides a broad framework in which programming 
must be carried out. The specific conununity also is con- 
sidered with its economic, social, enviz^onmental , and 
cultural needs and desires; its formal and informal power 
and social structure; its linkages among various organiza- 
tions; its potential for change based on past traditions 
and values. For' example, the socioeconomic status of people 
r ^eds to be considered— *are there large numbers of low-income 
people? Also, is it \ Dlitically feasible to shift resources 
to low-income programs? Does your personal value system 
say it*s important to do so? Are other community resources 
being used for low-income programs? 

Clientele. There *s a long tradition of basing programs 
on clientele needs. We all believe in and support this 
tradition. However, the question is; What clientele and 
what clientele needs? W^ need to look at clientele in terms 
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of subgroups within a community. Maybe we use Maslow's 
theory of needs as a framework for identifying priorities 
of various socioeconomic groups in the community. This 
could say to us that the focus for lower socioeconomic 
groups would be on physiological and social needs like 
nutrition or housing, while the higher socioeconomic groupings 
would focus on the upper part of the hierarchy — self- 
actualization. 

Political. Programming decisions are made within political 
structures. We recognize the influence of government at the 
town, county, state, and federal levels in establishing 
program priorities. Governmental units should be used in 
our decision-making process, but not become such a con- 
straining force that they overbalance all the other categories. 
There are times when program decisions aren*t politically 
attractive, but we need to take risks and accept responsibility 
for influencing the political structure at various levels 
so more rational decisions can be made. However, the question: 
"Is it politically feasible?" is always relevant. 

Organizational, organizational statements of philosophy 
and mission provide insights to many questions about 
priorities for programs and clientele. For example: Is 
the organisation supportive of innovations? What^s the 
organization's position in programming with controversial 
issues like family planning? What kinds of programs does 
the organization reward? How much .flexibility and persona?, 
input does the individual staff member have? What is che 
organization's position on programming with and for institu- 
tions and agencies that affect families versus programming 
with individuals ot> groups of individuals? 

Resources. There are many questions about resources 
that need to be answered in setting priorities: Do we have 
the quality and quantity of resources necessary to effect 
change through a program? Are they the right kind of 
resources? Are we employing new personnel to coincide with 
changing program priorities? 

Personal. We're interested in facilitating a self--* 
perception among staff that they're educational leaders. 
If we support this concept? then provisions for use of 
personal values and perceptions must be allowed in '':he 
decision-making process. 

The relationship among the categories must be considered. 
For example, you have to consider the political feasibility 
along with personal desires and needs of clientele in making 
a specific decision about programming with a difsadvantaged 
group. The categories can be viewed as constraining and 
driving forces interacting against one another in a decision- 
making process. The third way of viewing the use of these 



categories is as "screens" at which point a decision ma^ be 

made to go or not to go with a specific program becau?^ 

of a judgment based on criteria within one of the ca^.egories* 

Boyle, Patrick 6. "Criteria for Program Priorities 
Paper presented at a conference of Extension Home 
Economists Program Leaders 9 Washington, D.C. , November, 
1972 • 
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1*5 DEVELOPMENTAL EVALUATION 
ORIGIN Development of curriculum packages and prototype programs. 

ESSENCE Both the original design and the design as implemented 

must be carefully evaluated with immediate feedback into the 
design* 

USE Particularily useful to those developing pilot programs 

or packaged curriculua. However, also important in evaluating 
the "first run" of a program that will be repeated many 
times • 

STARTING Determine the purposes of evaluation and hov evaluation 

POINT fits j.nto the total development process. 

DESCRIPTION The first example described below comes from elementary 

and secondary education (Individually Prescribed Instruction 
Program CiPI] of the University of Pittsburgh Learning 
Research and Development Center). The second comes from 
the Canadian NewStart adult education program. 

IPI FORMATIVE Formative evaluation is the continual 

EVALUATION evaluation of all elements of a 

developing educational program as 
an aid to the development process. It*s concerned with this 
basic question: How can every element and operation in the 
program be examined so it contributes to its improvement? 
It has four sub-questions: 

1. What goals should the program achieve? 

2. What is the plan for achieving these goals? 

3. Does the operating pvogvcon represent a true implementa- 
tion of the plan? 

Does the program^ when developed and put into operation, 
achieve the desired goals? 

The four questions identi^ sequential steps in planning and 
developing a program. They must not only be examined in the 
evaluation, but also in relation to one another. 

Questions like the following are examined within the four 
steps. 
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Goals 

!• Are the statements actually goals? 

2» Are the stated objectives the real goals of the 
program? 

3* Are the goals worthwhile? 

Are the goals attainable? 

Plan of the Program 

1# Does the plan give promise of contributing to the 
achievement of the goals? 

2» Is the plan developed in sufficient detail? 

3» Will the plans and procedures be easily understood 
by the people who are to put them into operation? 

^0 Is it probable that the plan can be carried out? 

Program Operation 

1* What are the specific points to observe in 

operation analysis? According to the plan> what 
should the students be doing? What should the 
teachers be doing? What other personnel are 
involved and what observations should be made of 
their activities? 

2. Are the activities actually being carried out 
according to the plan? 

3» How can the operation be made to correspond more 
closely to the plan? 

^, Does a study of the actual operation suggest any 
modifications in the plan? The feasibility and 
effectiveness of a plan can only be determined 
after it has been studied in operation* 

Assessment of Results 

1» Do plans provide for assessing results? 

2. Are the assessment procedures reliable? 

3» Is the total evaluation process comprehensive 

enough to provide the needed total picture of the 
program result??? 
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4. What are the implications of the results for 
modifying the program? 

Formative evaluation ensures that each step in program 
development is done with care and assesses the quality at each 
step. Weaknesses discovered at any step have implications 
for modifying the proceeding level. Feedback is essential 
in formative evaluation. 

IPI evaluation also includes: 

Individual Pupil Monitoring. The regular and systematic 
evaluation of student achievement for adapting instruction 
to individual needs. 

Summattve Evaluation. The evaluation of the results produced 
by an educational program for purposes of making judgments 
concerning its value. 

SOURCE: Lindvall, C. M. , and Richard Cox. The IPI Evaluation 
ProgxHm* AERA Monograph Series on Curriculum 
Evaluation. Chicago: Rand McNally 8 Company, 
1970. 



NEWSTART EVAL^ The primary purposes' for developing 

UATION SYSTEM an evaluation system are: (l) to 

get information required so that 
course developers can make appropriate decisions to accept, 
modify, or reject developed components and concepts (formative 
evaluation), (2) to get information that^s required for 
dissemination activities (summative evaluation), and (3) to 
get information that will help the users in the installation 
of developed and tested course packages. 

The general scope of evaluation is indicated by: topic 
areae where deoieione are made, type of data baae used in 
the decision, eouroe of the data for the decision, and 
time of the decision. 

Areas Where Decisions Made. Decisions are made about 
policy, level of intervention, type of intervention, areas 
and agencies of intervention, dissemination practices, 
organizational structure, methods and systems of evaluation 
and measurement, theory and models, training methods, 
student progress, coach characteristics (ability, training, 
personality, style), course/curriculum content, and so on. 

Data Base. The type of data base on which these decisions 
are based can be divided into data from the implementere of 
interventione including such things as objectives of agencies; 
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social pressures, personal values, and prejudices of policy 
makers; professional judgments and opinions; priorities and 
standards; etc.; and data from the people T^^.ceiving the 
intervention including such things as attitude and per- 
sonality measurement, statistical results (descriptive 
statistics, difference tests, measures of association, etc.), 
standardized tests, skill levels of students, student employ- 
ment rates and incomes, demographic statistics on students 
and potential students, and so oii. 

Source of the Data* is similar to the data base, but involves 
more detail about the people on which the data are gathered, 
for example, funders, support organizations, communities, 
user agencies, present and potential students or clients, 
outside professionals and consultants, outside evaluators, 
project personnel (management, development, training, research, 
and support staff), and sometimes even abstract logical and 
rational considerations. 

Time of the Decision. This can range from very early 
policy decisions through the various stages of development 
of an intervention or program to decisions about where to 
implement programs. All of these aspects of the decision 
evaluation process nuast be considered in formulating a,, 
complete evaluation system. 

Formative/ aimnative evaluation integrates different methods 
into a total research study: direct observations create a 
global picture, surveys and standardized tests supply objective 
data to identify individual and group differences, case 
studies offer a connected sequence of events to help deter- 
mine and explain individual and group changes, the experiment 
provides a controlled test of the effectiveness of specific 
variables, and the overall research evaluation design provides 
for an interweaving of the various methods. 

A more specific modeling of the activity of evaluation 
includes : ' 

1. Determine decision areaa — both decisions that 
must be made and those that already have been made. 

2. Determine who the decision makere are. 

3. Determine information requirements ^^M\iat information 
is r'iquired by when. 

^. Determine methodology — source of data, instruments, 
how obtain, how process, how report. 

5. Prepare detailed prooeduree. 

6. Get approval of component development. 
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The model charts: (1) the kinds of evaluation 
information needed, (2) the stage where the information has 
primary or secondary use, and (3) the type of information — 
whether it's logical analysis, professional judgment, opinions, 
etc., or whether it's gained from empirical trials and measure- 
ment of effects on stidents. 

The following questions illustrate the kinds of information 
needed: 

1. How effective or efficient is the master or prime 
instructional strategy (individual prescribed 
instruction, group process)? 

2* How closely does the developed course reflect the 

concept expressed in the outline design specification? 

3, How closely does the developed course fit the parameters 
and bounds described in the outline design specifica- 
tion? 

How valid and complete is the course content? 

5. How closely do the instructional objectives meet the 
stated criteria? 

6. How realistic are the stated student entry char- 
acteristics? 

7. Lesson plans /curriculum guide and associated methods 
and materials: 

a* How closely do they meet the stated criteria? 

b. How closely do they result in student achievement 
of the instructional objectives? 

c. How effective and efficient is the student- 
instructor organization (specialist, coaches, 
one coach per group)? 

d. What additional or alternative methods or materials 
exist for achieving the instructional objectives? 

e* How effective and efficient is the student-coach 
ratio? 
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8, What^s the optimal sequence in each course and between 
associated courses? 

9. What effect does group composition have on outcomes? 

10. To what extent do the students achieve the instructional 
and course objectives? 

11. To what extent do the students retain and apply the 
skills, knowledge, and attitudes learned? 

12. How compatible are the instructional staff behaviors 
with the course objectives? 

13. How appropriate are the standardized tests used for 
the student population? 

14. How well do the administrative and support systems 
meet the needs of the potential users of the course 
•package? 



SOURCE: Lamrock, A. L., A. D» Smith, and P. W. Warren. 

' "Evaluation: Its Scope and fjystems for Ev^iluation 

Development." Paper prepared for Meeting of ' 
Research Directors of NcwStart Corporation, 
Ottawa, Canada, March 29-31, 1971. 
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1.6 MATERIALS EVALUATION 



ORIGIN 



Evaluation of new curriculum guides and audio-visual materials « 



ESSENCE 



Decisions among materials should be made using a firm set 
of criteria and an orderly process* 



USE Used whenever you*re selecting a packe^ged curriculum or an 

aid to use in a program, 

STARTING Determine the qualities needed in the item and the range 

PROGRAM of available products that may meet, the need. 



DESCRIPTION Although data gotten through questionnaires and interviews 

with teachers and students who are using the material is 
important input into the judgment, this approach depends 
primarily on professional judgment. Using a model of some 
type helps you identic the key aspects where comparisons 
must be made and avoid being swayed by highly motivating 
aspects of a particular program, 

"Trade-offs" must be considered carefully, A program 
may be strong in one area and weak in another. How does 
this compare with a second program that has different 
strengths and weaknesses? Which will you trade for the 
other? 

Here are two examples of approaches to materials evalaationt 
One deals with it as a process. The other outlines and 
wei^s criteria to be applied, 

TRADE-OFF AND One approach outlines the following 

COMPARATIVE COST activities: 

APPROACH 

1. Deaorihe the product to be evaluated as completely as 
possible. Description includes such things as intended 

. use, length, time involved in using, who the potential 
customers are. 

2. Evaluate the goala of the product. Are the goals 
appropriate? 

3. Clarify the point of entry of the evaluator. Which 
decisions are irreversible? Which can be influenced 
by the evaluator? 

Determine the kinds of tradeoffs involved. What other 
ways are there of attaining the same goals? What's 
- given up by accepting this particular product? 

O 

ERLC 
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5. Compare ooete of the product with costs of other 
alternatives. The comparison includes costs involved 
in handling, storing, repairing, and using as well 

as original cost of purchase. 

6. Make an intrinaia (seoondary) evaluation. Such 
evaluation might include technical quality, content 
evaluation, use of the uniqueness of the medium, 
feasibility of use. 

7. Make an outcome (primary) evaluation. Such evaluation 
might include learning rate and knowledge acquisition 
and retention. 

8. Form Qudgmentv and reaomendations. Recommendations 
based on the judgments may need to be specifically 
directed to potential purchasers, producers , and 

f unders . 

9. Stipulate airaumstances that might modi^ the con- 
clusions • 

10. Evaluate the evaluator. What might his biases be? 
What was over-emphasized? What was left out or . 
given inadequate attention during the evaluation? 



SOURCE: Glass, Gene V. "Educational Product Evaluation: 
A Prototype Format Applied-" Educational 
Researcher, I (January, 1972), 7-10. 

WEIGHTED CRITERIA • This technique calculates the 
APPROACH cost-effectiveness of alternative 

curriculum materials by a detailed 
breakdown and analysis of their components, quality, and 
cost. The model includes the following components and 
weights: 

I. Coverage (50). 

A. National, state, and college standards (30) • 

£• Qualitative comparison with standard materials (20). 

1. Scope i^). 

2. Detail (^). 

3. Accuracy ik). 
Clarity («^). 

5. Logic of sequence i^). 
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II, Appropriateneee (50). 

A. Utility ClO)^--type of job preparation. 

B. Difficulty (10) — ^for teacher, for student. 

C. Suitability of media to material (10). 

D. Suitability of media to student needs (10). 

1. Direction (5). 

2. Structure (5). 

III. Motivational Effeotiveneee (50). 

A. Teacher (20). 

1. Stimulation (6). 

2. Perceived utility (8). 

3. Demand level (6). 

B. Student (30) 

1. Stimulation (U). 

2. Perceived utility (10). 

3. Cultural relevance (14). 

4. Demand level (2). 
IV. Coat (50). 

A. Dollar cost (30). 

B. Time cost (20). 

1. Student (10). 

2. Teacher (10). 

When two or more alternative materials are being explored, 
the weights would be used in computing scores for each of the 
alternatives. The score would range from 0 to 200. However, 
in addition to comparing the total score, sub-ratings should 
also be considered* 



SOURCE; Crane, Peter, and Clark C. Abt. "A Model for 
Curriculum Evaluation." Eduoatiortal 
Technology^ IX (October, 1969), 17-25. 



98 



1.7 PARTICIPANT REACTION APPROACHES 
ORIGIN Adult education and extension. 



ESSENCE A variety of techniques including steering committees, 

analysis of attendance and re--enrollment records, and 
and end -of -session reaction forms are used to get 
participant reactions to programs. 

USE The information is used to modify programs to make them 

more attractive and valuable to participants. 



I STARTING Identify what you want participants to react to. 

POINT 



DESCRIPTION Almost every programmer has his own approach to examining 

participant reaction. Rather than presenting any one 
model, an overview of the approach will be given. There 
are two general approaches to getting reactions. In the 
first and most commonly used approach, the programmer 
specifies what he wants information about. In the second, 
he uses an open-ended approach and gets the participant to 
comment on what the participant sees as most important 
in terms of how he reacts to a program. 

When the programmer sets the parameters on what will be 
reacted to, he Visually concentrates on one or more of the 
.following: 

1. Content: newness, clarity, relevance, and importance. 

2. Format: timing, adequate involvement, interest 
holding, ease of learning, understanding what's 
expected, and suitability of techniques and approaches 
used. 

3* Faailitiee: location of meeting, room arrangement 
(see, hear, etc*), and auxiliary resources (meals, 
etc.). 

^. Teacher ^8 Perfomtanoe: style of teaching, organization, 
ability to relate well to participants, hindering 
mannerisms, ability to communicate, skill in involving 
others, etc. 

5. Peraeption of Value: what was learned,, what 
will be used, how important or useful, 
was it worth the time to attend, were objectives met, 
were expectations met, did it satis^ the need, and 
were underlying assumptions sound? 
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6. Follow-Up and Suggee tiorw for Future Programa: 
what more is needed on this sxibject, and what other 
programs are needed? 

7. Vccriation in Reaotiona Aoaording to Pet'^'^-f'^d Charaaterietioe : 
experience, position, age, educat' ^ first-timers t 
etc. 

When participants are left to react to programs without 
any structured guidance, they* 11 respond either in termi^ of 
what bothers them or in terras of what they think they should 
say. It's useful sometimes to ask how they describe a "good" 
program and then ask them to rate the program on those 
characteristics. Often the major points that concern them 
are: (l) the value, practicality, and use of what they get 
from the program; (2) how participation makes them feel 
(motivated to do something, happy and worthwhile, etc.); 
and (3) extent to which their interest is held. Details 
in terms of content, perf orm-^nce , etCj are important only 
to the extent that they affect how the adult enjoys participat-- 
ing and his feeling of getting something worthwhile from 
that participation. 

Systematically exploring participant reactions is especially 
important when you: 

1. Are doing something new— new clientele, new content 
area, or nev* teaching approach. Getting kinds of 
feedback that will help you determine how to sdapt 
and adjust the ne!^ becomes important. 

2. Know your clientele so well that you take them for 
granted. 

3. Have an uneasy feeling that things aren't going as 
well as they should, but you can't really put your 
finger on why. 

U. Are doing something that you feel is especially 
important and want to know how others react. 

5. Are doing something that ycu expect will be particularly 
difficult, conflict producing, valuable to some but 

not to others, etc., and you need to see how the .total 
group reacts. 

6. Know there's e. split in feeling, but don't know the 
numbers or relative strengths involved. 

7. Will be using the same resource people again and 
need evidence in hand to discuss changes in their 
contributions with them. 
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Although we're usually curious about how people react to 
rograms, just getting reactions is of little value unless 
you're willing to listen to what's being said and use 
participant reactions in making further decisions • 



SOURCES Steele, Sara M. "Exploring Participants Views of Programs." 

Material prepared for students in a course on Evaluation 
in Extension, University of Wisconsin-Madison, Spring, 
1972. 

Steele, Sara M. "Determining the Quality, Importance and 
Suitability of Programs," Madison, Wisconsin: University 
of Wisconsin, Department of Agricultural and Extension 
Education, 1970. 



ADDITIONAL Kropp, Russell P., and Coolie Verner. "An Attitude-Scale 

REFERENCES Technique for Evaluating Meetings." Adult Education^ 

VII (Summer, 1957), 212-15. 

Knowles, Malcolm. S. The Modem Praatiee of Adult Education. 
New York: Association Press, 1970, pp. 231-33. 

Axford, Roger N. Adult Eduaation: The Open Door. 
Scranton, Pennsylvania: International Textbook 
Company, 1969, pp. 195'-20if. 
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GROUP 2 

Evaluation of Program Parts 



OVERVIEW: 

What specific parts of a program should be evaluated? How do parts 
relate to one another and form systems? How do systems affect programs? 
Another way of patterning models for evaluation is to examine the program 
system and identic its key parts. This group of approaches provides a 
wide rango of ways in which the program system can be examined. 

EXAMPLES INCLUDED: 

# An exeaution-impaat approach for social policy makers to use in 
examining a program as a means of carrying out social policy. 

# A framework developed through analyzing adult education and select- 
ing aotuai aomponenta, 

fi A group of nmiagein^al ay stem • 

# A group of eot^io^organisationat approaches that emphasize human 
elements. 



GEMRAL ADVANL/JES: 

0 Programs are made up of things, activities, and people. Evaluation 
frameworks that identify the key elements, parts of a program, and 
their relationships are important in addition to frameworks that 
deal with stages and processes. 

^ Although the models differ in terms of what key elements they're 
talking about, all illustrate procedures in using a systems approach 
to evaluation. 



GENERAL CAUTIONS: 

0 It*s easier to anauLyze a system into its component parts than dt 
is to put it back together again and evaluate the effectiveness 
of the relationships among the parts. 

0 Success rests with identifying the most crucial from among the 
many diverse elements in a pro^^am* 
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2.1 EXECUTION-IMPACT APPROACH 



Social science research. 



The two major elements of this approach are aeeeeement of 
execution (finding out if the plan was carried out as it 
was expected to be carried out) and aeeeeement of the 
impaot of the program (what results it did produce). 

Although directed to social policy makers, it may be 
particularly helpful in examining programs carried out 
by many individuals or units working autonomously at 
different progium locations. 



Review the program plan. 



This approach is included in a description of how social 
science research can help social policy makers. The evaluation 
framework is part of a total framework that also calls for 
contributions of social research to the planning of the 
program. The need for good planning and careful desigyi as 
a requisite for expecting to find much impact is an important 
aspect of thu total model. In evaluation, the policy maker 
seeks to determine whether i 

1. The program was carried out according to prescriptions 
set forth in the planning and development stages. 

2. It worked. 

3. The expenditure of resources has been efficient 
compared with alternative means of achieving the 
same objective. 

The evaluation section of the framework is concerned with 
execution and impact. 

Assessment of the Execution. This includes: 

1. Determining whether the program has been directed 
at the appropriate target population. Eligibility 
criteria should be clearly stated and the program 
kept from bowing to pressures and diverting from 
the target clientele. 

2. Keeping records on who's being served. This record 
keeping must not interfere with program operation 
or be an undue burden on resources. 
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3. Examining the extent to which the needs of the population 
are met. 

4. Determining how well its actual operation conforms to 
" the plan for the program. This may involve examining 

the effect of various styles programmers use in carry- 
ing out the same program activities. 

5. Examining programs through direct observation by means 
of field studies. 

Assessment of the execution of a program is an administra- 
tive task. It ensures that the program that*s being tes-^ed 
actually was carried out as expected. It provides data for 
better preparing others to carry out similar programs. 

Assessment of the Impac**. Examines efficacy and efficiency. 

Efficacy. Determines whether the input changed the target 
group or environment in the direction expected, and whether 
such a change would occur again if the program were repeated. 

Efficiency. Refers to the improvement in terms of some 
measure of unit cost. Time, reduced need, and other similar 
factors may be used in calculating efficiency as well as 
monetary units. 

When two programs or approaches to programs are being 
compared and both look equal7.y effective, efficiency is a 
crucial determining factor. 

Freeman, Howard E., and Clarence C. Sherwood. SoaidL 
Reeetxtdh and Sooial PoHay. Englewood Cliffs, New 
Jersey: Prentice-flall , Inc*, 1970 • 



10^ 

2.2 ACTUAL COMPONENT APPROACH 



ORIGIN Analysis of adult basic education (ABE) programs nationwide. 

ESSENCE Evaluation is focused on: staffing reerui-hnent, collabora- 

tion^ in-aervice education^ goale, instruction, and the re- 
lationahvp among them. Each is seen as having inputs, pro- 
cesses, and outcomes. Intended practice for each component 
is compared with current practice to identify gaps for pro- 
gram improvement. 

USE Particularly useful in identifying crucial programming 

elements and monitoring and strengthening their functioning. 

STARTING List the components of a program that are seen as crucial 

POINT to the functioning of the program. Set the criteria for 

selecting those that will be evaluated. 

DESCRIPTION Two criteria were used in select.lng the six components . 

More components can be added if desired. These six were 
selected because: (1) they showed ways ABE program people 
interacted interdependently to make the program function 
and (2) program people were most concerned about them. 

Detailed guides were developed for getting information 
on intent. Current practice is determined through question- 
naires developed for administrators, teachers, students, 
and co-sponsors as well as classroom observation forms. 

Analyzing incongruities between intent and reported 
practice, the evaluation approach systematically examines 
qualitative differences in program expectations of ad- 
ministrators and supervisors, teachers, students, and 
among these three groups. Analysis of these data pinpoint 
real and potential trouble spots. Intent is also compared 
with current practice for each component and discrepancies 
in expectations among individuals and groups are identified. 

Each of the six components is examined in terms of inputs^ 
process^ and outcomes. Specific aspects of the six compon- 
ents that were identified for examination are listed here. 

Component 

for Examination Inputs Processes Outcomes 

Staffing Administrative Searching Characteristics 

time Selecting Competence 

Budget Placing 
allocations 

Selection 
criteria 
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Component 

for Examination Inputs 



Recruit-- 
ment 



Collabora- 
tion 



In-service 
eduoation 



Administrative 

time 
Budget 

allocations 
Target 

population 

Administrative 
time 



Money 
Time 

Number of 
staff 



Processes 

People involved 
Methods used 



Working 
relationships 



Methods used 



Outcomes 
Numbers 

Characteristics 
Interest 



Recruitment 

Allies 

Facilities 

Competence 



Goal 
setting 

Instruction 



Legal mandates 



Staffing 
Recruitment 
Collaboration 
In-service 
education 
Goal setting 



Deciding on 
priorities 

Teaching- 
learning 
transaction 

' Individualiza- 
tion 

Mat8rial3 
used 



Objectives 
Criteria 

Persistence 
Achievement 
Further education 
Occupational 
development 



The components are closely interrelated. Relationship is 
most apparent when the outcome of one component is a major 
input into another component such as when the outcome of 
recruitment is the number and characteristics of students — 
an important input into instruction. The central component 
is instruction. The other five components all are inputs 
into instruction. 



This evaluation approach: 



1. Is designed to help people compare what they're doing 
with what they want to be doing as a basis for making 
changes, so the results will be more satisfactory. 



2. Is based on a naturalistic systems model in which 
evaluation activities are part of the feedback that 
facilitates a more effective relationship between 
inputs, processes, and outcomes to encourage ongoing 
organizational development. 
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3. Is organized to: (a) focus on an initial set of 
program components of concern to practitioners, 
(b) allow for the selection and modification 

of data collection instnunents, and (c) facilitate 
the use of those instruments and procedures as 
examples in preparing evaluation materials for 
additional components. 

4. Combines internal evaluation for familiarity and 
commitment to use results, with external evaluation 
for impartiality and awareness of outside resources 
and standards. 

5. Includes procedures to incorporate current evaluation 
activities and data — such as enrollment, achievement, 
and financial reports to the state — and ^ittendance 
reports that are so often used as indicators of 
teacher effectiveness. 

6. Contains explanations of procedures designed to 
encourage people to participate in the evaluation 
activities and to use the results for program 
improvement . 

7. Contains items that might be adapted for an exclusively 
external evaluation, even though it wasn't designed 
for this purpose. 

Knox, Alan, Jack Hezirow, and Gordon Darkenwald. Evaluating 
Adult Basic Education: A Strategy and a Manual. New 
York: Columbia University, Teachers College, Center for 
Adult Education, 1973 [pre-publication draft]. 
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2.3 MANAGERIAL SYSTEMS 



ORIGIN 
ESSENCE- 



Industry and the military. 



Evaluation models drawn from management theory and practice 
give greater emphasis to the role of management in successful 
programming. They deal with system and interaction in the 
system as they would in getting products produced in industry. 



USE 



Particularly useful in more efficiently managing programs < 



STARTING 
POINT 



Analyze the system, 
relationships. 



Determine its components and their 



DESCRIPTION Administrative decisions are made on the basis of the 

judgment of thf^ administrator and/or his consultant. Previous 
experience and a logical mind play a key role in these 
decisions. They are often instant and static* Managerial 
decision making is also based on the judgment, experience, 
and logical naind of the decision maker, but is guided by 
data that are systematically collected and analyzed. 

Management is the process of planning, operation control-, 
supervision, and production engineering in which the decisions 
are data based. Two of the broad purposes for evaluation 
interface with the management concept: 

1. To provide quantitative information or systematically 
collected data for consideration in decisions leading 
to the better conduct of an operation or to increase 
functional utility of a product under development. 

2. To provide information or systematically collected 
data for consideration in decisions dealing with the 
adoption or support of a product or activity. 

There are many relevant frameworks in managerial science 
and operations research. A few have been included as examples 
to indicate how systems are analyzed and how such analysis 
is used. 



PROGRAM EVALUATION 
AND REVIEW 
TECHNIQUE (PERT) 



This technique, often known by its 
key initials PERT, involves a process 
by which: 



1. Projects are broken into specific task components! 



2« Specific tasks are graphically presented so that 
sequence and interface are clearly identified. 
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3. Responsibility for completing each task is clearly 
established* 

4. Time estimations and deadlines are established. 

5. Key steps and pivotal activities are identified. 

The tasks, relationships, people responsible, and dead- 
lines are diagrammed in flow chart form. Diagrams may be 
simple straight lines if only sequential relationships 
exist. However, most involve branching to show the sub- 
activities and other related activities. A flow chart is 
developed by starting with the deadline for the completed 
task and then working backward until all of the tasks have 
been charted. Three items are usually indicated: 
optimistic (0) time of completion, most likely time (M), 
and pessimistic (P) time. 

The following criteria help in deciding whether to use PERT: 

1. Does a specified objective exist? Can the accomplishment 
of this objective be determined? 

2. Must some schedule date or deadline be met? 

3. What's the degree of project complexity? As the 
complexity of a project increases, the need for PERT 
grows. 

4. Does a degree of uncertainty exist about the definition 
of some of the program elements? It may be particularly 
helpful in complex situations; it's less apt to be 
needed in routine and standardized situations. 

SOURCES: Roman, Daniel "The PERT System: An Appraisal 
of Program Evaluation Review Technique." In 
'Program Evaluation in the Health Helds^ 
Herbert C. Schulberg, Alan Sheldon, and Frank 
Baker, eds. New York: Behavioral Publications, 
Inc., 1969, pp. 243-53. 

Ryden, Einar R. "PERT? What Is It?" Keeping 
Current, Washington, D.C.: Federal Extension 
Service-U. S. Department of Agriculture, March, 1967. 

Cook, Desmond L. Progrm Evaluation and Review 
Technique: Applioatian in Eduoabion. Washington, 
D.C.; U.S. Government Printing Office t 1966. 
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ORGANIZATION AS This approach suggests that managerial 

A TOTAL SISTEM technology can be best used when the 

organization has been designed as a 
total system. The whole needs to be analyzed into its com- 
ponent parts. 

Organization Is a Process with Inputs and Outputs, in 

business , the input is resources and the output is economic 
welfare • 

Management Represents the Control or Feedback Mechanism. 

Given an expected output » the actual output is measured 
against it and, if a difference exists, a problem is indicated. 

The management or control unit is separated into a series 
of parts to provide adaptive capability. One type of adaptive 
capability is the ability to change inputs to maintain the 
expected output. 

The adaptor unit within the control system has an input 
analyzer (for example, market analysis) that indicates change 
and an identifier that indicates the state of the organization 
or any of its subprocesses at a given time (ext^mt to which 
resources are presently fully used). 

Both the input analyzer and the identifier feed information 
to the decision maker. He applies a set of rules that enables 
management to make decisions about changes in input, output, 
and/or process to adapt to changing needs and conditions* 

The organization is apt to have several major subprocesses. 
In industry, they include marketing, production, finance, and 
personnel. Each has its own adaptor. Each is sensitive 
to and reacts to both the other subprocesses and external 
stimuli. 

Learning capability in the form of a designer is added to the 
adaptive system. The learning capability is the ability of 
.system to redesign itself and learn from past mistakes to 
improve system performance. 

The design proceae includes: (1) the originating problem, 
(2) a model, (3) specifications, (4) pilot simulation 
studies, and (5) field operations. The model includes the 
following elements: input, output, process, system logic, 
and information requirements. 

Inputs are of three kinds: the input that enters the 
system and on which the process is to operate, environmental 
inputs that affect the process, and instrumental or control 
inputs that modify the operation of the process or the 
process itself » 
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Design cxn.tetn>a are rules used to evaluate the acceptability 
of designs. The most widely used are measurability , feasibility t 
optimality, reliability, and stability. MeasUrability is the 
system *s ability to evaluate its performance. Feasibility 
indicates whether the model will operate as planned. All parts 
of the system must be integrated and mutually consistent for 
feasibility to occur. Optimality is concerned with the best 
choice. However, the optimal choice may not be feasible. 
Something that will work comes first and then optimality is 
dealt with. 



SOURCE: Young, Stanley. "Organization As a Total System." 

In Emerging ?c ;teme of Administrative Aocomtdbility s 
Lesley H. Browder, Jr., ed. Berkeley,, California: 
McCutcha.'i Publishing Corporation, 1971, pp. 311-30. 

MACRO SYSTEM Evaluation is defined as the process 

MODEL of first identifying and then quantifying 

the relationships between student in- 
puts and educational outputs, given a constant financial input and 
controlling for effects of external systems. 

Student Inputs. Refer to the nature and characteristics 
of the students entering the program to be evaluated. 

EcUicational Outputs, include (l) changes in students as 
a result of the program and (2) impact of the program on home, 
community, other progreuns, etc. 

Financial Inputs. Refer to the financial resources avail- 
able to carry out the program. 

Mediating Factors. Are the descriptive characteristics 
(personnel, organization, instructional design) of the way 
the financial inputs are used in the program in combination 
with the student inputs. 

External Systems. Mean the social, political, legal, 
economic, and other systems outside of the school, formal and 
informal, that encompass the program, have impact on it, and 
are in turn modified by the outputs of the program. 

The model is pictured with student and financial inputs 
entering the system, being exposed to the mediating factors 
and then exiting as student and other outputs . This occurs 
within the context of the effects of exteraal systems. 

Student outputs are seen as more than scores of students 
on academic achievement tests* They' include changes in 
attitude and practice and the students ability to deal with 
real-life situations* 
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The changes that occur through education, have social, 
political, and economic implications. The very nature 
of the external systems is altered by changes in student 
outputs. 

SOURCE: Alkin, Marvin C. Tomrds cm Evaluation Model: 
A by sterna Approach. Working Paper No. tf. 
Los Angeles: University of California, Center 
for the Study of Evaluation of Instructional 
Programs, August, 1967. 

SYSTEM APPROACH System analysis is a process , a 

TO GOAL SETTING rational mode of approaching complex 

problems, a tool that helps in sound 
decision meJcing. Developed to help administrators better 
relate the systems approach to program budgeting, the frame- 
work has the following main parts: 

1. . Systems definition, structure, and design. 

2. System design process. 

3. Defining the boundaries of the system and identifying 
how those within the system judge the performance of 
that system. 

Need for goal integration and method to obtain it* 
5. Goal-setting process • 
Key ideas within the parts include: 

Systems Definition, Structure, and Design. A system is 
defined as a set of elements whose relationships are designed 
to accomplish a desired goal. This particular framework 
deals with the school system. Tne elements of the school 
system are identified as: philosophy , goals ^ objeotivees 
inputs and/or resources; outputs; the environments the programs; 
the agents^ decision makers^ and management; and measures 
of effectiveness. 

Programs are defined as the processes by which inputs are 
converted into outputs. In the school context, programs are 
the alternative types of curriculum that could be devised 
to achieve the objectives. Programs can also be broken 
down into activities, projects, components, and/or elements. 

The element of agents^ decision makers^ and management 
stresses the role of people in management. A management 
system is an organization and as such involves agents and 
decision makers that make it work. It's important in system 
analysis to account for all the decision makers across or- 
ganisational boundarieb who contribute to the performance of 
a particular function* 
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System Design Process. System design is a prcblcm-solving 
process starting with the needs of the client. Needs are 
translated into specific goals and objectives that are ex- 
pressed in terms of measures of effectiveness. These measures 
serve as standards of achievement against which the results 
of alternative courses of action can be compared. The 
evaluation of outcome;3 will be made in the context of a value 
model that represents' the relations^4ip among the important 
variables of the system. Results will be compared to the 
cj^iginal design and a new level set dependent on the degree 
of success or failure achieved. 

Defining the Boundaries of the System. Education means 

different things to different people. Teachers and parents 
may not have the same perceptions. Actors within the system 
who hold expectation about the system include: teach rs, 
non-teachers, parents, students, community, nation, ana 
university and higher education. Decisions are affected by 
how administrators set boundaries for inclusion or exclusion 
of people with concerns about education. Conflicts often occur* 
For example, teachers may be concerned with "best" education t 
while taxpayers want average education commensurate with 
reasonable taxes. 

Goal Integration. The administrator must get general 
agreement on the general philosophy and broad goals from 
all the groups in the system. The goals and objectives, and 
the ability to attain them, are influenced continuously by 
the contraints on the system which, in turn, is influenced 
by the cultural environment it operates in. 

Goal -Setting Process. The goal-setting process starts 
with a statement of general philosophy or mission, then 
moves through general goals, subgoals, and objectives. To 
be useful an objective must be (l) quantifiable or ob- 
servable, (2) definite, (3) measurable, (U) obtainable, and 
(5) in conformation with the organization's goals. 

SOURCE: Van Gigch, John P., and Richard E. Hill. Ueing 

Syetems Analysis to Implement Cost Effectivemss 
md Program Budgeting in Education. Engiewood 
Cliffs, Mew Jersey: Educational Technology 
Publications, 1971. 

PROGRAM CONTACT A- system approach is a managerial 

SYSTEM effort taken to initiate an operational 

program that^s designed specifically 
to generate a smooth, effective, and efficient flow of infor- 
mation from those providing it to those in need of it for 
decisionnnaking purposes. 



113 



Within the Extension program planning format, the system 
approach includes a structure, an interacting complex of 
Extension and research people, facilities, and procedures 
designed to generate an orderly flow of pertinent informa- 
tion for dissemination and use as the basis for decision 
making in specified responsibility areas. This information 
is collected from both intra- and extra-program sources. 

The increasing volume of information flowing from 
university research units is likely to continue. If Extension 
is to realize its mission of extending university knowledge 
off campus, programs must be planned and organized to 
accommodate and facilitate this increased flow. Information 
roust first be evaluated to see that it*s: (l) suitable to the 
audience *s needs, (2) sufficient in quantity and quality to 
help in problem solving, and (3) provided in time for the 
recipient to use it. 

The following programming system is proposed: 

Ph,ibe I— Program Formulation, includes: (i) information 

external to the specific program, (2) research needs according 
to established priorities and tempered by current research 
capabilities, and (3) stored information awaiting the 
activation of a new program. In the beginning stages, program 
formulation may only consist of assembling a series of ideas, 
concepts, and policies. 

Phase II— Decision Phase. This phase is made up of four 
subphases: (1) program development, (2) program strategy, 
(3) program dissemination strategy, and (U) program promotion 
strategy Program development involves explicit recognition 
of audience characteristics and needs and consideration of 
alternatiye program approaches that will increase effective- 
ness. Program strategies consider the socioeconomic and 
institutional environments and restraints. 

Dissemination strategy necessitates the study of: (l) exist- 
ing channels of information distribution, (2) types of in- 
formational outlets, (3) size and general characteristics of 
the setting clients live in, and (4) capabilities of the 
disseminators. Promotional sti-ategy includes media efforts 
to precede, accompany, and follow the program. All of the 
strategies would be combined into one workable package. 

Phase III-*«-Execut1on Phase. 

Phase IV~Eva1uat1on of Program Results, This must be a 

joint effort between researchers (source of information that's 
disseminated) and extenders # Historically research personnel 
have had no way of determining: (.1) if their efforts proved 
of practical iinportancei (2) if their knowledge was extended 
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in a timely and efficient manner, (3) if their "laboratory 
findings" were valid (or invalid) in general application, 
and (U) if a redirection of their efforts or of those of 
Extension personnel were warranted. 

SOURCE: Duft, Ken D* "Systems Planning for Extension." 

Journal of Cooperative Exteneion^ VII (Fall, 
1969), 168-78. 

MANAGEMENT Cooperative Extension adopted a 

INFORMATION nationwide management information 

SYSTEMS (SEMIS) system a few years ago. The system 

plays a dual role in the conduct of 

. Extension programs. 

SEMIS (state Extension Management Information System) consists 
of planning, activity reporting, and some type of evaluation. 
One part of SEMIS in absence of the other two is incomplete. 
However, each part when analyzed may offer help in decision 
making and/or resource allocation. When these data are used 
together, they make a base for administrative and professional 
decisions . 

SEMIS provides the professional a system for categorizing planned 
effort and for evaluating the effectiveness and efficiency of 
those efforts. The two roles Siren* t mutually exclusive, but 
complement each other. 

Each decision is based on the best information available, 
whether it be hearsay, observations by untrained individuals, 
or factual data collected through a formal system. Obviously, 
the latter source of data is superior. Therefore, the 
management function is concerned with the total efforts 
planned or expended to accomplish a purpose and the benefits 
derived from the Extension effort. 

The data provided through a formal system such as SEMIS 
are provided by each person who planned the program, carried out the 
tasks, and evaluated the results of the work. He knows better 
than anyone else what he did, when he did it, how he did it* 
He also has the competency to evaluate its effectiveness. 

SEMIS isn't designed nor is it capable of serving as a 
sxibstitute for professional judgment. It's a way of 
recording and transmitting these judgments. It provides 
a means for categorizing planning, activity reports, and 
evaluation to be compatible with all other staff member's. 
SEMIS provides two things: (l) a factual record of when, how, 
where, and with whoiu each activity was planned and conducted 
and (2) the contributions the specific professional and his 
activity made to the overall program. 
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SOURCE: EMIS Data Definition Document^ Fieoal Year 1974. 

Washington, D.C.: Extension Service-USDA, 
Extension Management Information System. 

EVALUATING One approach to evaluating a decision 

DECISION requires the construction of a decision 

MAKING matrix. The matrix consists of: (l) a 

problem requiripg a decision* (2) a list 

of alternative decisions, and (3) a generic list of criteria 

used to consider each alternative decision. 

The generic list often contains items classified into ^various 
types. Three classifications are: (l) technologioal—'is the 
solution workable with the available resources, (2) eooiopeyoho-^ 
logical — is the solution likely to maintain morale, and (3) 
eoonomiO'-'^oes it fit within the economic environment and policies 
that affect that environment? The generic criteria must be 
developed into specific operational criteria expressed in such 
terms as resource load, results, and value of results. 

The cells of the matrix are weighted as either positive , 
negative, or neutral. Actual decision making is observed 
and the number and types of cells actually used are noted. 
For example, how many alternatives are considered? How many 
and which criteria are used? 

Adequate analysis takes into account both the final decision 
itself and the strategy used. 

SOURCE: Hesse ling, P. Strategy of Evaluation Reeearoh. 

The Netherlands: Royal Van Corcun Ltd., 
pp. 217-34, as cited in A Strategy for 
Evaluation Design^ Casper F. Paulson, Jr., ed. 
• Oregon State System of Higher Education, 1970, 
section VII, pp. 22-2U. 

ADDITIONAL Lange, Robert R. "Evaluation As Management Methodology." 

REFERENCE In A Strategy for Evaluation Deoign^ Casper F. Paulson, Jr., ed. 

Oregon State System of Higher Education, 1970, section VII. 
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2.4 SOCIO-ORGANIZATIONAL SYSTEH5 



ORIGIN Social science theory. 



ESSENCE These models focus on the human rather than the non-human 

components of programming. They're concerned with facilitating 
changes that will improve programming. 

USE Programs aren't done by machines. Success lies with the people 

involved. Models that examine social and psychological dy- 
namics give good information on how programs can be improved. 

STARTING Identification of the human and social system elements that 

POINT are most relevant to the success of programs. 

DESCRIPTION The quality of the program and the extant of its accomplish- 

ments depends on the quality, organization, and capability 
of the agency that offers it. 

The purpose of evaluation isn't to prove but to improve 
and to facilitate activities necessary for improvement. 
Changes included in new programs and changes in old programs 
that are recommended by evaluation reports have to be made 
by people. Bringing about such changes needs as careful 
planning as does the actual programming or its evaluation. 
Change can be facilitated by evaluative processes that take 
adequate advantage of the system and factions involved. 

There are several social and social-psychological models 
that might be adaptable to program evaluation. Attention 
is just beginning to turn to these approaches and few of 
them have been thoroughly adapted and tested. Three models 
developed for other purposes are included as examples of types 
of frameworks that have potential for evaluating and improving 
programs . 

SOCIA^ SYSTEMS Although designed for analyzing a 

MODELS social system, social systems 

may have a good deal to contribute 
to evaluating how an agency programs successfully. One of 
the most widely known models is that proposed by Loomis. 
It has three key kinds of ingredients: (1) etemsntes 
(2) proae9B0B, and (3) requieitee for eoaial aoHone. 

Elements, include ends or objectives, norms or standards » 
facilities, power, rank, sanction, status roles, knowledge, 
and attitudes. 
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Processes, include communications 9 socialization* in- 
stitutionalization, boundary maintenance, systemic linkage, 
and social control. 

Requisites for Action, include time, territoriality, and 
space • 

Although you may examine the same ingredients, the relation- 
ships that are brought into focus in program evaluation may 
be different. For example, prime attention would be given 
to the elements of standards and norms and ends and objectives 
as -envisioned by a social system (programming agency , govern- 
mcr.t bureau, or cliente group) with the relationship of power, 
rank, sanction, and status and role examined in relation to 
the other two elements. The effect of the social system 
processes both on the program being evaluated and on the 
evaluation need to be considered. For example, communication, 
boundary maintenance, systemic linkage, social control, and 
institutionalization may all affect both the program and its 
evaluation. 

SOURCE; Loomis, CP. Social Syatema: Eeeaye on Their 
Pereietenae and Changes • Princeton, New 
Jersey: Van Nostrand, 1=60. 

ORGANIZATIONAL When applied in evaluation, the 

MODELS starting point isn't the mission and 

goal of the organization, but a working 
model of a social unit that's capable of achieving a goal. It's 
a model of a multifunctional unit and includes both the functions 
direct to the program and the maintenance and support functions 
essential to the system. 

It deals with such questions as how close does the organiza- 
tional allocations of resources approach an optimum distri- 
bution under these conditions and with this mission? Either 
a lack or an overabundance of resources in relation to given 
thrusts may be dysfunctional. 

It also concerns itself with feedback mechanisms. 

SOURCES: Etizioni, Amitai. "Two Approaches to Organizational 
Analysis: A Critique and a Suggestion." In 
Program Evaluation in the Health Fielda, Herbert 
C. Schulberg, Alan Sheldon, and Frank Baker, eds. 
New York: Behavioral Publications, Inc., 1969, 
pp. 101-20. 

Schulberg, Herbert C, and Frank Baker. "Program 
Evaluation Models and the Implementation of 
Research Findings." In Program Evaluatian in the 
Health Fields^ Herbert C. Schulberg, Alan Sheldon, 
and Frank Baker, eds. Kew York: Behavioral 
Publications, Inc., 1969, pp. 562-72. 



MOTIVATION Motivation helps to explain human 

MOdEh behavior. In education, concern is 

both with motivation within (pro- 
fessional) and the motivation without (the layman or client). 
In each instance there is input, motivational proaeeeee^ 
and output. 

Input. Input for lioth the client and the professional 
include: needs, goals, value systems, and abilities* 

Output. For the client, it's participation and behavioral 
change. Tor the professional, it includes effectiveness, 
efficiency, integration, and morale. 

Motivational Processes. These, which convert the input 
into output, have been grouped into three categories: 
organization, grox4p integration, and personal. Organiza- 
tion includes philosophy, rew6U?ds, and training; group 
integration includes goal setting, leadership, leaders/group 
members relationships, and formal and informal groups. 
Within personal is: achievement, recognition, communication, 
^responsibility, and participation. 

The model identifies evaluation criteria to be used .related 
to the various processes. The following are selected examples: 

Organization: Philosophy. Does the organization 
recognize the existence of individual's subconscious motives 
that influence actions? Does the organization strive toward 
helping people develop adequate self-esteem by treating the 
person as an individual? 

Integration: Goal Setting. Are the goals clear, precise, 
realistic, and internalized? Have individual goals been 
established and are they in' line with the organization's 
goals of efficiency and effectiveness? 

Integration: Fonnal and Informal Groups. Are the norms 

and purposes of the informal group appropriate for accomplish- 
ment of the organization's and individual goals? 

Reward: Achievement, is achievement and level of 
aspiration of the individual recognized as a vital aspect 
of motivation? Does the organization provide outlets for 
the individual's pride of accomplishment? 



SOURCE: Lewis, Robert B. "Motivation Model for Extension." 

Journal of Extensions X (Winter, 1972), 23-24. 



GROUP 3 

Evaluation—Kinds of Data; Types of Activities 



OVERVIEW: 



What kinds of data do you deal with in evaluation? What types of 
activities are involved? This group of approaches "model" the kinds of 
data used in evaluation or the kinds of evaluative activities that can 



EXAMPLES INCLUDED: 

# The caimtenwee of evaluation identifies three major categories 
of data about programs and four types of data that can be used 
in forming conclusions. 

# The ayetem role model categorizes data in terras of program elements. 

# The means-enda hieporohy shows how various kinds of common evaluative 
data on process and results relate to each other. 

# The when^-to-'do^it^youTeelf aontinuum systematizes the kinds of 
data-gathering activities usually carried on by program units and 
suggests wiien the teacher-programmer may need to involve an expert. 



• Understanding kinds of data and data-gathering activities helps you 
better landerstand evaluation. 

^ Such imderstanding also helps you be more selective in investing 

resources in getting data and managing data activities more efficiently. 



GENERAL CAUTIONS: 

0 Data always have to be related to the purpose of the evaluation. 
There are many kinds of data that could be collected, but not 
all of them are crucial to each evaluation. 

^ The average prograwner can't keep up on all types of data-gathering 
techniques; therefore, knowing when to call for help and where to 
turn is important. 



be employed. 



GENERAL 



ADVANTAGES: 
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3.1 COUNTENANCE OP EVALUATION 



ORIGIN 
ESSENCE 



Public school education. 



Evaluation data are categorized as being either deecriptive — 
intents and obeervations — or judgmental* Both descriptive 
and judgmental data are gathered about anteaedenta^ tvaneaatimBs 
and outoomeem Examining oontingenciee and oongrueneiee 
among the sets of data is an important part of evaluation. 



USE 



Particularly helpful in understanding program dynamics 
and in selecting the roost crucial data for a particular 
evaluative purpose. 



STARTING 
POINT 



Develop the rationale for evaluation. Why is it being done? 
Who will use it? For what purpose? 



DESCRIPTION Education is best improved through adequately dealing with 

its complexity and dynamic nature. The components of the 
program are parts of an interdependent system. An evaluation 
model should identic the components of evaluation. Although 
these components may in some cases be the same as progi^am 
components, the evaluation model considers them only as 
they're a component of the evaluation. The evaluation should 
focus on the parts of the program that will produce the most 
program improvement for the evaluation dollar. 

Educational evaluation has both an informal and formal side# 
The informal includes '^asual observations, implicit goals, 
intuitive norms, and subjective judgments « The formal 
includes check lists, structiu?ed visitation by peers, controlled 
comparisons, and standaj^dized testing. Neither gives full 
accord to the dynamics of education • 

Evaluation includes two acts — deeanption and judgment^-^ 
both are essential* It seeks generalizations about education*^ 
al practices. Even though an outside evaluator may be un- 
willing to judge, he must include both judgmental and des- 
criptive data in his work. 

The kinds of data to be included are viewed in a matrix 
(two-dimensional table) type of arrangement* 

Horizontal Side. The horizontal side of the matrix 
includes the three major categories of data to be examined: 
antecedents, tponsaetione^ and cutaomes. These are generic 
labels that cover a variety of real-life items • 



Antecedents • Are conditions existing before the teaching 
that nay affect the outcone. For exanpl«t the status of the 
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student at the start— his past experience, aptitude, interest, 

level of knowledge, and skill — is an important antecedent. 

The term may be equatable with input if input encompasses stude.^t 

and community inputs as well as the input of the educational 

system* 

Tranaaetione • Are encounters of the student with teachers, 
other students, authors of texts — the succession of engage- 
ments that make up the process of education. 

Outcomes. Are evidence of the impact of instruction on 
students, teachers, and others; changes in abilities; 
achievements, attitudes, and aspirations of students; wear 
on the equipment; effect on the school; cost. Outcomes 
include both those evident and existing at the end of the 
session and applications, transfer, and releaming that 
occur later* Outcomes are the consequences of educating — 
immediate and long range, cognitive and connotative, personal 
and community-wide. 

Vertical Side. This dimension of the matrix describes the 
two kinds of deaoHpHve data and the two kinds of judgmental 
data that may be relevant to each of the horizontal categories. 

DeetsTvptive data. Include: 

Intente. Are descriptions of the planned-for environmental 
conditions, the planned-for teaching methods and content 
coverage, and intended student behavior. The collective 
descriptions should provide a priority listing of all the 
things that are intended to happen. Emphasis is given to 
what teaching as well as what learning is intended. Part 
of the information comes from the rationale for the program. 

Obeervatione . Are descriptions of what actually did occur 
related to the antecedents, transactions, and outcomes. This 
category of data can include direct observations. It also 
includes the realm of data about transactions and outcomes 
secured through the iise of tests, check lists, and other 
instruments • 

The two basic ways of using descriptive data are: 
(1) looking for aontingenaiee among antecedents, transactions, 
and outcomes and (2) looking for oongrmncy between the 
intended and the observed. 

E3q>loring the degree of congruence between intended and 
real (as measured and/or seen) and the contingencies between 
the antecedents, transactions , and outcomes creates a much better 
understanding of the effectiveness of plans and implementation. 
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Judgmental Data. Two approaches are: (1) dbaotuie 
etandavde as reflected by personal judgments and 
(2) relative standards as reflected by characteristics of 
alternative programs. 

Absolute Standards. Indicate levels set for antecedents, 
transactions, and outcomes. Data provide information on the 
judgment of the extent to which the standards are met. This 
evaluative activity involves identifying possible standards 
and weighing or assigning relative importance to each set of 
standards considering how much attention should be paid to each 
reference group (students, curriculum experts, etc). 

Relative Standards. Are similar to absolute standards, but 

use other programs as tte base of identification. The base 
of comparison may be either the intended or the observational 
data. In the first case, you're using judgments about the 
progToun as it's planned. In the latter, you're using judgments 
about the progr-am as it was actually carried on. Both types 
of standards may include several characteristics. As a result 
of judging the program antecedents, transactions, <iud/ov 
outcomes, a coiq}osite rating of merit can be developed to use 
in making further decisions. 

Use of descriptive data helps you understand programs, but 
unless some type of judgmental- data is also used, conclusions 
about worth and value aren't present and the evaluation hasn't 
been completed. 

Often only parts of the total countenance are used in a 
specific evaluation. The framework aids in deciding what 
categories and kind of information will be examined in the 
evaluation. For example: 

!• Is it to be primarily descriptive, primarily judgmental, 
or both descriptive and judgmental? 

2. Is it to emphasize the antecedent conditions, the 
transactions, or only the outcomes; or a combination 
of the three; or their functional relationships? 

3. Is the evaluation to emphasize congruence between 
intended and observed? 

Is it to focus on one program or to compare two or 
more programs? 

5. Is it intended to help in deciding between programs 
or to help in developing op improving a program? 
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SOURCES Stake, Robert E. "The Countenance of Educational Evaluation." 

Teachers College Record, LXVIII (1967), 523-40. 

Stake, Robert E. "A Research Rationale forEPIE The 
EPIE Form, I (September, 1967). 

Stake, Robert E., and Terry Denny. "Needed Concepts and 
Techniques for Utilizing More Fully the Potential of 
Evaluation." In Eduoational Evaluation: Net) Rolee 
and JHf^ Meanss Part II , 68th Yearbook of the National 
Society for the Study of Education, Ralph H. Tyler, ed. 
Chicago: The University of Chicago Press, 1969, chapter 
XVI, pp. 370-90. 



Stake, Robert E. "Objectives, Priorities and Other Judgment 
Data." Reuieta of EduoaHanal ReBearchj XL (April, 1970), 
181-212. 
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3.2 SYSTEM ROLE MODEL 



Adult education. 



Key components of the model are: contexts inpute^ pToaeeSj 
activity s outcome judgment ^ and application of findinge. 

Particularly helpful in identifying the range of specific 
data that might be examined about a program so that the most 
relevant can be selected. 



Describe the program as a system. Iderti^ the roles played 
within it. Determine the relationship of evaluation to the 
system and the roles. 

The general purpose of evaluation is to improve the educational 
program by facilitating judgments about its effective- 
ness based on evidence. Propi?ammers and administrators should 
consider the following points as they design evaluation systems: 

Evidence. Systematic and continuous program evaluation 
procedures should provide more adequate evidence and improve 
the soundness of judgment. 

Benefits. The extensiveness of the evaluation procedures 
depends on the importance of making sounder judgments. 
There should be a balance between the costs of evaluation 
and the benefi.ts received. 

Frequency. The frequency with which evaluation data 
is collected depends on the aspect of the program that 
is being assessed and the anticipated use of the results. 

Feedback. Evaluation's major function is providing 
continuous internal feedback to enable adjustments in the 
ongoing program. 

Commitment. Those who are affected by the evaluation 
should participate in the evaluation process so that the 
likelihood of their using the results will be increased* 

Objectivity. The outside evaluation specialist can help 
to increase the objectivity and validity of evaluation 
procedures . 

Standards. Appropriate standards for comparison should 
be established. Both the relative performance of similar 
programs and absolute standards of excellence should be 
considered. 

/ 
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Relevance. Data that are highly relevant to the intents 
and objectives should be collected and analyzed. 

Values* The process of making judgments should include, 
in addition to data, appropriate values, consensus, and continuing 
commitment . 

The system role approach includes seven components that 
are identified from system and role theory and the analysis of 
past research and evaluation* Each component has a specific 
set of evaluative activities. 

Context. An understanding of the setting in which the 
program takes place and the influence of that setting on the 
program. Examining the program rationale is an important 
part of the setting. The rationale within which a continuing 
education program is developed and evaluated has three parts: 
history, current demands and constraints, and expectations 
about future development. 

Five related activities include examining the setting, deciding 
on the most important evaluation emphasis, selecting evaluation 
models and procedures, and identifying the context in which 
the evaluation will occur. 

Inputs. Include both what was intended and what actually 
occurred. Participants, materials, teachers, and administra- 
tors are all inputs into the program. Input characteristics 
of participants include: 

1. Biographical — age, sex, travel experience, work history. 

2. Status-^-prestige, Ic^'el of work, income level, level 
of living. 

3. Ability-- performance on intelligence and aptitude tests. 
U. Personality--attitudes , self -concept . 

5. Roles— present family situation, type of employment. 

6. Needs— gaps between present and desired circumstances. 

Categories of information suggested for materials evaluation 
include : 

1. Content goals— what knowledge should be gained? 

2. Performance goals -^^hat should the learner be able to 
do with what he has learned? 

3. Teacher requirements — what should the teacher know to 
use materials? 
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Participant expectations— what should learners want to 
accomplish? 

5* Community factors — characteristics of the community that 
influence effectiveness of material. 

6. Activity -Rescript ion of the intended and the achieved 
inputs into the programs* 

Staff inputs should be considered in terms of roles — in- 
structional, administrative, and support. Among characteristics 
to be examined are subject-matter competence and ability to 
relate to participants. Inputs that go beyond characteristics 
are support inputs like amount of encouragement and in-service 
education. Basic activities include determining the nature of 
important inputs and then describing the quantity of these 
Inputs . 

Examining inputs: 

1. Makes it easier to plan to acquire the inputs. 

2. Helps with the planning of intended learning activities 
and outcomes that- can be achieved, based on the .in- 
tended inputs. 

3. Provides the basis for interpreting the learning 
activities and outcomes that actually occior. 

Process. Description of the intended and the achieved 
transactions that bring together inputs to produce outcomes. 
The heart of the program is the teaching-gleaming process » 
Interactions in programming take place in four settings — 
individual, temporary group (classes), organizational 
(in-service), and community. 

Intended processes are descriptions of the ways inputs 
should interact to produce outcomes. The achieved processes 
are those that actually occur. Plans are a major source of 
the intended processes. Information on what actually occurs 
can be gotten from analyzing interaction, using trained 
observers, rating scales completed by students, etc. 

Activity. One way of analyzing process is how well it 
transforms resources to, maintain stability and stimulate 
change. Examples of some of the things to be examined 
include: goals and policy, program development, teacher 
selection and supervision, learner selection, teaching- 
learning transaction, learner support and advisement, support 
staff selection, administrator selection, maintenance of 
personnel, adaptation ana change, facilities and equipment, 
materials, coordination and coramunication, financing. 



Outcomes. Vary from inmediate stages that describe 
direct changes in learner's knowledge, skills, or attitudes 
to more remote stages where outcomes are stated in terms of 
benefits to the community. The key question is whether it's 
reasonable to expect these inputs and transactions to produce 
the intended outcomes. 

Major categories of outcomes include: 

!• General impact — ^progress participants make, proportion 
who complete, adequacy of progress* 

2. Participant satisfaction — do studerts feel they have 
goTten enough? 

3. Content mastery — test scores such as those on standard 
equivalency exams. 

Personality — ^improved self-image and greater social 
awareness • 

5. Work related— increased income, higher employr.ent rate, 
reduction of welfare rolls. 

Evaluative activities include describing the intended and 
the achieved outcomes of the educational program. Major 
sources of intended outcomes includ-^ expectations of par- 
ticipar,::s, teachers, administrators, and policy makers 
associated with the program. 

Comparison of intended outcomes and achieved outcomes 
(descriptions of both) may indicate that adjustments should 
be made to raise achievement, to adjust expectations, or 
both. 

Judgnients. Judgments about the extent the intended was 
achieved in terms of inputs, process, and outcomes have 
been made earlier as a part of analyzing the logical sound- 
ness of the program plan. Conclusion about needed changes 
in activities or expectations or both should have been 
formed . 

The next stage in mak5 judgments focuses on results. 
It involves internal and t -.ternal comparison. Internal 
judgments involve the extent to which the actual inputs and 
processes contributed to the achievement of the outcomes 
and whether the outcomes (benefits) compare well with the 
inputs (costs). External comparisons analyze the results 
of the specific program against external standards. 
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Application of findings. Conclusions should be used 
to improve the program. Reports should focus on the ques- 
tions and concerns that staff and others have, and be 
designed to result in application. 

Guidelines for handling results include: 

Validity. Evaluation procedures that are well planned 
and implemented to produce valid results increase the 
potential use of the findings. 

Corrmmioationa . The "lults of evaluation efforts 
should be communicated to ,hose who can use them — learner, 
teacher, administrator, and policy maker — in a form they 
can readily understand. 

Commitment* Involve those who can use the results in 
such a way that results are valid and there's commitment 
to use them. 

Timing* Results should reach those who can use thern 
during a time period in which their use is feasible. 

ImplicatianB * The major implications of the findings 
should be included in both* the evaluation report and the 
discussion of It. 

Time. Ways should be found to allocate time for the 
study and use of evaluation fii^dings. 

Aeaiatanae. Technical help should be available both 
for additional analysis and interpretation of findings. 

SOURCES Knox, Alan B. Program Evaluation in Adult Bacio Education. 

Tallahassee, Florida: Florida State University, 
Adult Education Research Information, Processing 
Center, 1971. 

Knox, Alan B. "Continuous Program Evaluation." In 
Adminiatrati'^n of Continuing Education, Nathan C. 
Shaw, ed. Washington, D.C.; National Association 
for Public School Education, 1969. 

Knox, Alan B. "Improving Urban School Adult Education: 

An Analysis of the Adult and Community Education Programs 
of the Washington, D.C. Schools." New York: Columbia 
University, Teachers College, 1967. 
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3.3 MEANS-ENDS HIERARCHY 



ORIGIN 
ESSENCE 



Cooperative Extension. 



The model shows the relationship of seven levels of 
evaluation data: irtputa^ aotivitiea^ people inoolveds 
reaotione, "KAS" ohcmgee^ change in praoHoee^ and 
ultimate reeulta. 



USE 



Particularly helpful in situations where several kinds of 
input and result data can be generated and where the evaluator 
has to decide which to examine, or, if examining several, 
needs a rational way of relating them to each other. 



STARTING 
POINT 



Start with a decision about what information needs to be 
collected about the program, process, and program results. 
The model then helps you understand how this information 
relates to other information. 



DESCRIPTION The means-^nds hierarchy deals with process and results. 

It^s aimed at improving program development. It can be 
applied both in constructing agency-*wide strategies an4 in 
designing specific evaluation projects. 

It can be applied through evaluative research, evaluative 
studies, and the "criteria" approach to evaluation. The 
three used in tandem is suggested as one systematic strategy 
for agency-wide evaluation. 

The model includes seven levels of e valua tion: inputs, 
activities, people involved, reactions, "KAS" chaK^es (know-^ 
ledge, attitude, and skill), application of learning or 
"practice change," and results achieved, that is, degree 
of problem solution. 

Level I~InputS. Description of the amount of effort, 
e.g., time and money spent. Simplest and easiest data to 
get. They're needed for cost-benefit analysis at a higher 
level. 

Level I I --Activities. Descriptions of the kinds and 
frequencies of educational activities performed in the project. 

Level III— People Involved. Descriptions of numbers of 
participants reached and nattire of balance in terms of 
social characteristics. 

Level IV— Reactions. Descriptions of what people say about 
the activities— whether they see they ^ re valuable, their like 
or dislike of the quality and quantity of activities. 



Level V— "KAS" Changes- Data on (l) whether changes 
have occxirred in specific knowledge, attitudes, and skills; 
(2) the direction of such change; and (3) the distance or 
extent of change • 

Level VI --Practice Change • Explores whether people have 
used their knowledge, skills, and attitudes and whether 
desirable patterns of behavior have resulted* 

Level VII— Results Achieved • This category examines what 
consequences the changes in behavior have brought to people 
and conununities • There's an underlying assumption that each 
of the lower levels do in fact have a bearing on bringing about 
the final results. The data in each of the seven levels can 
be related to each other. For example, inputs can be com- 
pared with "KAS" changes or with practice change or with 
results achieved* Or, recipient and reaction data can be 
compared with input or with results achieved. 

Other propositions are: 

1. Any evaluative question within the hierarchy is 
related to, and has implications for, all the rest 
of the questions. 

2. Evaluation at the lowest levels of the heirarchy is 
rather meaningless for impact evaluation unless we 
either know or make assumptions about the answe. 3 to 
"higher" evaluative questions. 

3« If a higher-level evaluative question is answered 
positively through research, the achievement of 
lower-level objectives can be interpreted as progress. 

^« Evaluative studies at the higher levels should be 
stressed* These evaluation studies can help provide 
a basis for interpretation of less costly, lower types 
or evaluation. 



Bennett, Claude F. ''Theory and Strategy for Evaluation of 
Knowledge DiBeeminabion.'^ Washington, D.C: Extension 
Service-OSDA» Program, and Staff Development, 1973* 
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3*4 WHEN-TO-DO-IT-YOURSELP CONTINUUM 



ORIGIN 
ESSENCE 



Cooperative Extension. 



Data-^gathering activities are grouped into five main 

categories: habitual but unorganized, eimple guides^ 

reporting by atudente or teacher, poet-- or pre- 

and poat'-teating, and experimental deaign uaing control 

or comparative group. Although program personnel are 

responsible for carrying out most of the activities, help 

should be gotten from specialists for more complicated activities. 



USE 



Particularily useful in en5>hasizing the shared roles between 
program personnel and evaluation or studies specialists. 



STARTING 
POINT 



Identic the type of data activities needed in the evaluation • 



DESCRIPTION The major purpose of evaluation in education is to 

determine the effects of teaching under giv<:n conditions on 
the knowledge, attitudes, and behaviors of those being 
taught to provide a basis for improving, justifying, or 
discontinuing the teaching activity. 

The major focus of evaluation is to try to determine 
what kind of individual emerges from the learning experience 
to which he has been exposed. 

The continuum of types of data-gathering activities is 
as follows: 

Habitual But Unorganized 

1. Habitual but unorganized introspections of the 
teacher relative to the teaching sittiation. 

2. Panel or staff group who jointly and informally 
review a teaching situation without a list of 
questions or itsms to guide the review. 

Simple Guides 



Teacher's rating by means of a list of questions 
or items of his teaching relative to what he did 
and effects on those being taught. 

Observations of a teaching situation by an . 
individual or by those taught, guided by a list 
of questions or items. 
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5. Observations of a teaching situation by a panel 
guided by a list of questions or items. 

Reporting by Students or Teacher 

5. A system used by the teacher for reporting the 
learning of individual students. 

?• Analysis of an educational experience by students 
using a well-designed post-questionnaire* 

Post- or Pre- and Post«test1ng 

8. A post-questionnaire or test that determines the 
knowledge, attitudes, and skills of students 
resulting from one teaching situation. 

9. A questionnaire or test that determines knowledge, 
attitudes, and skills of students resulting from 
one teaching situation. 

10. A questionnaire or test before and after more 
than one teaching situation (often several as in 
a program extended over time) that determines 
the knowledge, attitudes, and skills of students 
resulting from the exposure; desirable to be 
accompanied by an input study. 

Experimental Design Using Control or Comparative Group 

11. A questionnaire or test before and after an 
educational experience (either one or more exposure) 
measures knowledge, attitudes, and skills; a 
similar pre- and post-testing of a matched control 
group; a study of the educational input. 

12. Experimental study using a control or comparative 
group in which the educational experience (either 
one or more exposure) including methods is planned 
for studying what happens in students; before and 
after questionnaires or tests are used; a study of 
the educational input. 

The framework suggests that the program personnel can handle 
Activities 1 and 2 by themselves. They may want some help 
from studies specialists for 3-5 depending on the complexness 
of the task and the programmers background. Type 6 activities 
are developed jointly by program personnel cind studies specialist, 
conducted by program personnel. Activities 7-9 are developed 
jointly, but conducted by either program personnel or studies 
specialist. Activities 10-12 are planned and conducted by 
studies specialist with assistance from program personnel. 



SOURCE 

ADDITIONAL 
REFERENCES 



Alexander, Frank D. "A Critique of Evaluation." Joiamal 
of Cooperative Extemion^ III (Winter, 1965), 205-12. 



Steele, Sara M. Progranmer'e Reeponeibility in Examining 
Program Effeetiveneee. Evaluation Series, Madison, 
Wisconsin: University of Wisconsin-Extension, 
Division of Program and Staff Development, October > 1972. 
[Identifies program personnel responsibility regardless 
of whether help is gotten from evaluation or research 
specialists . ] 

Kirkpatrick, Donald L. "Evaluation of Training." In 
Training and Development Bandbook, Robert L. Craig 
and Lester R. Bittel» ods. New York: McGraw-Hill 
Book Company t 1967 » pp. 87-^112. 



GROUP 4 

Evaluation Processes 



OVERVIEW: 

How do you evaluate? What are the processes involved? Another way 
of "modeling" evaluation is in terms of processes. This group of approaches 
focuses on the processes of evaluation. The first three deal with basic 
processes used regardless of what's being evaluated. The other three present 
specific evaluation processes. 

EXAMPLES INCLUDED: 

0 The appraieal model enn>hasizes the role of professional judgment. 

0 The data management framework provides a guide for dealing with 
quantifiable data regardless of what's being examined. 

# The natural prooeee approach shows how appraisal and quantitative 
approaches can be coid>ined in aiding decision making. 

# Monitoring evaluation shows how you can use evaluation during 
programming to keep the program on track. 

# Improvement evaluation uses appraisal to stimulate program improvement. 

# TraneaQtional evaluation emphasizes the human relationships involved 
in facilitating change. 

GENERAL ADVANTAGES: 

0 Although specific procedures and activities vary, evaluation relies 
on certain constant basic processes. Good understanding of these 
processes can speed evaluation. 

# Processes that involve program personnel in a positive way are 
particularly important in getting evaluation used. 

GENERAL CAUTIONS: 

0 Processes should be kept as simple and unobtrusive as possible. 

# Simplicity should be viewed from the participants' eyss rathei? 
than from those of the evaluator. 
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4 • 1 APPRAISAL MOO^:ii 



ORIGIN 



Accreditation bodies and educational agencies responsible 
for examining the programs of local units. 



ESSENCE 



This approach emphasizes profeeeional judgment. Judgments 
are made by an expert, team of experts, or team of program 
peers — from within or without the system — who examine the 
program and form conclusions and recommendations. Criteria 
guides are used in examining the program. 



USE 



Particularily useful when immediate feedback and interpretation 
are needed and where interaction with program personnel 
facilitates the use of evaluation. 



STARTING 
POINT 



Identify the purpose of the appraisal and the purpose 
of the program that's being appraised. 



DESCRIPTION This process is used in accreditation, in reviews by state 

departments of public instruction, and in the comprehensive 
reviews of Cooperative Extension. AppraiBot ie an act of 
Judgment in which the judging impliee both a oriterionr^-a 
standard of earn eort^-^cotd a pertinent deeoHption of what 
being done. The criterion and the observation must deal 
with the same thing. 

Appraisal involves the following aativitiee: 

1. Specifying the purpose of the appraisal. 

2» Determining who will serve as the appraisers. 

3. Establishing the purpose of the program. 

4. Selecting or developing a set of criteria. 

5. Identi^ing the aspect or aspects that will be evaluated, 

6. Recognizing and understanding the implications of the 
assumptions that are being made when program aspects 
and criteria are chosen. 

7. Amplifying the criteria so that they become a detailed 
statement of the kinds of observations that need to 

be made* 

8. Developing a plan of action for making the observations 
(getting data a variety of ways). 



9. 



Developing » modifying » selecting » and using techniques 
of observation* 
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10. Determining number of observations and procedures for 
increasing validity 5 reliability, and objectivity. 

11 • Recording, interpreting, and sumnarizing the observations. 

12. Establishing bases to which the observations can be 
related (that is, selecting norms and standards as a 
base for interpretation). 

13. Making the conclusions of the appraisal known. 

The quality of the appraisal rests on the expertise of the 
appraisers and on the quality of the criteria used. Cvitevia 
constitute definitions of what is seen as "good," Criteria 
define what characterizes a good program. They express values 
or interpretations of what's good. Because values differ, 
there may be several criteria alternatives. What may be 
seen as good at one time and place may not be valued in the 
same way in another. 

Criteria must be consistent with the educational philosophy 
that's being followed. They postulate values that should be 
achieved. They must be so formulated to permit making 
pertinent descriptions. They must be useable. This often 
means a greater precision in definition than is first thought. 

Success of appraisal rests partly on answering the following 
criteria questions: What criteria? How defined? How applied? 
How limited? This model culminates in judgments and conclusions 
of people instead of, or in addition to, systematically 
produced data. 

It's also important to understand the OBBvanptionB involved 
in the evaluation. There are usually at least two kinds: 

1. Assumption of the purpose of education. 

2. Assumption of correlation or relation. 

The other key element is obaeTVOtion. Observations include 
both describing and interpreting what's observed. 

The aspects of a program that are subject to appraisal 
fall into four categories: 

1. Flans and purposes--^rganizational, administrative, 
and instructional objectives and plans. 

2. Resources-^Hnaterlal rasources (such things as 
furnishings, and AV equipment) and personnel. 

3. Processes^dministrative, supervisory, and instructional. 



4. Effects — ^results in terms of students, staff, and 
community* 

Although appraisal is often thought of as primarily percep- 
tion and interaction in on-site visits, it can also include 
review and interpretation of data about a program. The 
appraisal method is sometimes criticized for its lack of use 
of objective data. However, this may be more a fault of how 
it is operationalized rather than with appraisal itself. 
Data can be built into appraisal syst&ms. 



SOURCE Harris, Chester H. "The Appraisal of a School — Problem for 

Study." Journal of Educational Reaeardh^ XLI (November* 
19U7), 172-82. 



ADDITIONAL Herzog, Elizabeth. Some Guidelines for Evaluative Research. 
REFERENCES Washington, D.C.: U.S. Department of Health, Education, 

and Welfare, Social Security Administration, 1959. 

[ParticulcU^ly good section on criteria.] 

Checklists for Public School Adult Education Program. 
Circular No. 356. Washington, D.C.: U.S. Office of 
Education, Federal Security Agen<^, n.d. 

Honnold, Kobert E. et al. "Comprehensive Reviet^.^* 
Haterials prepared by a Program and Staff Development 
Task Force, Extension Servica-^USDA, 1972. 
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4.2 DATA MANAGEMENT 



ORIGIN 
ESSENCE 



Educational research # 



The processes of getting, processing, and reporting the data 
used in evaluation have been presented in many ways. The 
framework accompanying the program decision model is particularly 
comprehensive in that it both outlines the crucial activities 
and categorizes some of the specific tasks where assumptions 
are often made that must be tested for validity in the 
evaluation setting. 



USE 



The focus on the validity of the assumptions made in the 
selection of sampling, treatnient, measurement, and analysis 
procedures is particularly helpful. 



STARTING 
POINT 



Start with a cleeu? analysis of the data needed for the 
evaluation. 



DESCRIPTION The major elements involved in data management include" 

(1) delineating the information system^ (2) obtaining the 
information, (3) providing the information, and (4) evatmting 
the evaluation. 

Delineating the Information System, includes defining 

the system being evaluated, specifying decisions inherent in 
the program, developing a statement of evaluation policies, 
and clearly outlining the assumptions upon which the study 
is based. 

Obtaining the Information. This second step includes 

collecting, organizing, and analyzing data. 

Three steps in collecting evaluation data are: (1) identify- 
ing sources of information, (2) developing the proper instru- 
mentation, and (3) describing the conditions under which the 
data will be collected. 

The organization stage of obtaining the information 
consists of deciding on the unit of organization, storage 
and retrieval requirements, and quality control procedures 
followed after data collection. 

When the data have been collected and organized, the 
evaluation proceeds with the actual analysis. The first 
step is identi^ing the unit of analysis, which is determined 
by the level of the decision involved, the information 
required in making that decision, and the sampling unit 
employed. The basic tools of analysis include prediction, 
comparison t description t and explanatlon-^-^the assumptions 



ERLC 
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of the approach employed determine the analysis, statistics, 
and reporting form used. 

Providing the Information, Timing, frequency, and 
methods of these reports are determined by the audience 
for whom the evaluation is being prepared and the decisions 
within the program structure they relate to. 

The first step in developing the report defines the 
audience it*s intended for — decision makers, decision 
influencers, etc. This is relatively straightforward. 
But, the second task — depicting the reporting levels — is 
more difficult. Reporting levels can be dichotomized in 
terms of micro versus macro levels to define the level of 
detail of the information reported to the individuals within 
the programs and agencies. Consequently, evaluators must 
determine the audience for their reports, and accordingly 
produce them at the appropriate levels of generality or 
specificity, 

A third aspect at this stage ^involves establishing the 
reporting setting, developing the content, and choosing 
the proper media to communicate the findings. 

The information must be disseminated to the proper 
individuals. This involves; first of all, establishing 
channels for transmitting the reports and identifying the 
hierarchy to be followed in presenting the results. Re- 
ports should go to the administrator and the staff, before 
making the information available to funding agencies or the 
general public. A second consideration in disseminating 
reports is developing a procedure for publication. Abstracts 
may be provided to the appropriate news media, or where the 
information is unique, copyrighted to ensure protection 
over the use of the findings . 

Six criteria should be kept in mind if evaluative data 
are to be informative to the receiver: 

Relevanoe. The data are useless if they don't relate 
to the purpose they were collected for. The criterion of 
relevance asks whether the purposes actually have been 
achieved, 

ImpoPtanoe. Data shovdd be culled so that the most 
important information is emphasized. The person organizing 
the information must use what the user considers significant 
as well as his own view of what*8 important. 

Scope. Data can be truet but not the whole truth. Scope 
ensures that weeding for perceived relevance and importance 
doesn^t distort the data* 
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Credibility. Users must trust the source of the data. 
Credibility rests partly in the relationship between the 
data handler and the user of the data* 

Timlineee. The information must reach the user when 
he needs it and/or is in a position to deal with it. Its 
value decreases if it's available too late or if it appears 
too soon. 

Pervaaiveneee . Do all of the people who need to know 
about the evaluation get the information? 

Evaluating the Evaluation. The final element in this 
model is applying evaluative tools to the evaluation study 
itself. Since the audience will be weighing the quality 
of his study, the prudent evaluator carefully examines 
his work against the criteria on which it will be judged. 

Assumptions* The exploration of assumptions underlying 
various techniques and testing whether those assumptions 
are appropriate in the evaluation activity is an important 
element in designing evaluation. Four basic types of 
aeewnptione are of concern to evaluators: sampling, 
treatment, measurement, and analysis. 

Sampling Aeeumptione. Include: (1) parameter: of the 
population, (2) social structural characteristics of the 
population, (3) parameters to be tested, (U) objectives of 
the analysis > (5) representatives of the req^uired sample, 
(6) precision which the sample affords because of the 
selection method employed, and (7) assun^tions based 
on budgetary and resource limitations and precision relating 
to efficiency of sample selection methods for reducing 
uncertainty. 

Treatment AaQumptions. Include statements about internal 
and external validity. Internal validity assumptions relate 
to: (1) history of events between the pre- and post-test 
that may affect performance, {2) maturation of the subjects, 
(3) learning effects from the first test producing an impact 
on the results of the second, (U) instrumentation in which 
changes in calibration of measuring instrument or changes 
in the observers may produce changes in the obtained 
measurements, (5) effects of statistical regression, 
(6) biases from the selection of respondents for comparison 
groups, (7) experimental mortality ,i and (8) selection-maturation 
interaction in which designs are mistaken for the effects of 
the e)q>erimental variable. 



External validity assumptions include: (1) congruence 
between population characteristics and those of the sample, 
(2) ecological validity, including both an explicit descrip- 
tion of the independent variables assuming interference 
with subjects receiving two or more treatments, (3) Hawthorne 
effect, (U) novelty effects, (5) effect of the experimenter, 
(6) pr'3-test sensitization, (7) post-test sensitization, 
(8) interaction of history and treatment, (9) measurement 
of dependent variable required for generalization, and 
no) interaction of the time of the measurement and treatment 
effects . 

To establish limits of reliability in the treatment phase, 
the evaluator also should include assumptions about the 
additive effect of treatments, differential effect or treat- 
ment constancy, identical treatment of subjects, identical 
treatment of groups, random assignment of treatments, 
multiple treatments being avoided, and minimization of the 
interactive effects of the treatment with control variables. 

Measurement Aaaumptions . Are another integral part of this 
stage of an evr:iluation design. They should deal with sxxCa 
basic questions as: (1) correspondence between sets of 
axioms about objects and numbers, (2) representativeness of 
the sample, (3) scalers compatibility with the phenomena 
being measured and analysis being employed, (U) validity of 
the instrument, (5) whether the scale of meas\..'-Jient is 
conceptually or operationally defined, (6) reliability of 
the instrument, and (7) objectivity of the instrument or 
technique. This information helps the evaluator control 
sources of error and gives the clientele of the evaluation 
a clearer picture of the limitations of the study. 

Analytic Aaatanptiona. state the intentions of the evaluation 
and outline the use the descriptive, comparative, explanatory, 
or predictive information will be put to. Analytic assump- 
tions are concerned with the correspondence of statistical 
techniques with the properties of the data to be analyzed. 
Ass*'.mptions should be specified since statistical procedures 
ai't designed for or derived from them, ^ 

On the basis of assumptions about sampling, treatment, 
measurement, and analysis, the evaluator can develop a 
general pictm j of the kind of design or model that will 
organize and communicate there data roost effectively. More 
specifically, these may include mathematical models, work 
breakdowns, PERT organizations, or an application of the 
critical path network techniqu.^* 
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SOURCE Phi Delta Kappa Nationel Study Committee on Evaluation. 

Educational Evaluation and Decieion Making. Itasca, * 
Illinois: F. E. Peacock Publishers , Inc., 1971, 
chapter 6 and pp. 28-30. 



ADDITIONAL Caro, Francis G., ed. Readinge in Evaluation Reeearch. 
REFERENCES New York: Russell Sage Foundation, 1971. [In particular, 

see chapter 1 for an overvie*^ of the results of recent 

experiences . ] 

Corey, Stephen M. Action Reeearch To Improve Sohool 
Praoticee. New York: Columbia University, Teachers 
College, 1953. [Emphasizes action person doing research 
in addition to the professional researcher.] 

Suchman, Edward A. Evaluative Reeearah* New York: Russell 
Sage Foundation, 1971. 

: ; Weiss, Carol H. Evaluation Reeearch. Englewood Cliffs, 

New Jersey: Prentice-Hall, Inc., 1972. 

Freeman, Howard E., and Clarence C. Sherwood. Social 
Reeearch and Social Policy. Englewood Cliffs, New . 
Jersey: Prentice-Hall, Inc., 1970. 

American Institutes for Research. Evaluative Reeearch: Strategiee 
and Methods. Pittsburgh, Pennsylvania: American Institutes 
for Research, 1970. 

Gottman, John M. , and Robert E. Clasen. Evaluation in 

Education. Itasca, Illinois: F. E. Peacock Publishers, 
Inc., 1972. 

Furst, Edward J. Constructing Evaluation Instruments. 
New York: David McKay Co., Inc., 196^. 

Kaplan, Abraham* The Conduct of Inquiry. San Francisco: 
Chandler Publishing-Company , >19§^r-; 
lln particular, chapter. on values t3 

( 
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4.3 NATURAL PROCESS APPROACH 



ORIGIN 
ESSENCE 



Extension education. 



This attempt to provide a generic model of program evaluation 
that can be adjusted to evaluative needs emphasizes three 
major elements of evaluation: ariterias evi^danoBs and 
judgment. It emphasizes improving the kind of evaluation 
that program personnel do naturally rather than replacing 
such activities with elaborate formalized procedures. 



USE 



Useful in improving ongoing everyday evaluation and in helping 
program personnel understand evaluation. 



STARTING 
POINT 



Start with how the individual usually evaluates. See how the 
accuracy and efficiency of the natural process can be improved. 



DESCRIPTION Evaluation assesses the value of programs and programming 

activities. There are three major elements in evaluation: oriterias 
evidences and judgment • Evaluation takes place when judgments 
are made. Judgments are improved when they* re made by comparing 
evidence about the aspect to be judged with criteria for what 
should be. 

Criteria. A criterion is a measure against which something 
can be judged. It may be a rule, a standard, a norm, an object 
or a condition, or behavior that*s considered to be "good," 
"ideal," or of "high merit." It's a description or image of 
what a valuable (suitable, high quality, effective, important, 
and/or efficient) program is like. Criteria form the basis 
for interpreting information and developing judgments. They 
identify the values held by those involved. 

Sets of criteria exist for judging quality of meetings, 
workshops, and group interaction. There are fewer criteria 
statements that deal with overall value of an endeavor. 
Such statements are essential to setting priorities and 
assigning overall value to competing program requests. 
( Criteria identify the aspects on which alternatives will be 

compared. 

Clearly understanding and trying to meet a reliable 
set of criteria for programming can do a great deal to improve 
programming efforts regardless of whether those criteria 
are used in any formal type of evaluation. Criteria tell 
us What we should be doing. 

Evidence. Evidence is an indication or an outward sign. 
In evaluation evidence is: actSf words ^ numbers » observa- 
tions » or things that provide a sign or indication. 
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It provides proof of the extent to which the quality we're 
examining is present. It's the accumulated pattern of 
information that provides a picture adequate for judging 
the extent to which criteria have been met. Evidence is 
information accepted as descriptive of what's directly 
germaine to the judgments involved. Use of research-type 
techniques may improve the quantity and quality of evidence. 
Regardless of the extensiveness of the data procedure, the 
evaluator must be* able to assess the effects of limitations 
in the evidence. Few data are completely valid and reliable. 
The amount of attention to systematic exploration must follow 
a cost-benefit analysis. Where is the payoff from increasing 
objectivity going to be great enough to balance the costs 
of work involved in getting that objectivity? Conversely, 
where will the harm be sufficiently great due to inaccuracy to 
require a greater investment in the evaluative process? 

Judgment. Judgment is the part of the evaluation process 
in which alternatives are weighted, comparisons made, various 
conclusions considered, and worth or value assigned to what's 
being evaluated. Judgment is a mental operation involving 
coii5)arison and discrimination through which values and 
relationships are formulated. Evidence and criteria help in 
forming sound judgment, but neither constitute judgment. 
The quality of evaluation rests primarily with the ability 
of people to use criteria and evidence in forming sound 
judgment • 

Evaluation is an everyday process used by most people. 
That process can be improved by increasing the information 
base and degree of system in the procedures used. However 5 
for the evaluation to have "payoff," it has to be internalized 
by those involved. Therefore, the challenge is helping a 
process that must be subjective (within the mind) become 
more accurate. 

Procedures based on this model must be designed specifically 
for given situations. Generally they include: 

1. Identi^ing the purpose of the evaluation. (If it's to 
be 5,nput into decision making, what are the alternatives 
that need to be compared?) 

2. What crucial judgments must be made to complete the 
purpose? 

3. What criteria are germane to those judgments? 

. What type .of evidence is needed? How "pure" must it be? 

5. What is the most efficient source of such evidence? 

6. How does the evidence compare with the criteria? 
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7. What are the resultant judgments? 

8. What do they mean in terms of the purpose of the 
evaluation? 

. 9. What will recommendations or decisions based on those 
judgments mean to those involved? 

10. If they're to result in action, how can that best be 
accomplished? 

The natural process can use any of several criteria categories 
as a source of criteria-- for example, appropriateness, contact, 
effort, effectiveness, overall value* It can be applied 
to any of the programming processes, activities, or stages. 
However, greatest value is usually secured from good evaluation 
of plans and preparation before implementing the program and 
kind of evaluation during the program that has immediate 
feedback and chance to improve the program while it*s in 
operation. 

Evaluation (the assessment of value) provides important 
input into three kinds of decisions: decisions inherent in 
the development of programs, decisions inherent in the. 
improvement of programs, and decisions inherent in the fate 
of the program and programmer. " The amount of attention 
to program components other than results depends on the kind 
of status that the program is in. For example, evaluation 
of context, input, and process are particularly important 
when the program is in a development or improvement status and 
there's no question but that it will continue. Then 
result data are important in improving program operations. 
However, if the fate of the program and/or programmer is 
at stake, then evaluation of components other than results 
are secondary to examining results. Some attention is 
important however so that result data can be accurately 
interpreted. 

Steele > Sara M. "Program Evaluation — A Broader Definition." 
^owmxl of Exteneion^ VIII (Summer, 1970), 5-17. 

Steele, Sara M. "A Contemporary Concept of ;?rogram Evaluation." 
Madison, Wisconsin: University of Wisconsin » Extension, 1972* 
[Mimeographed] 
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4.4 MONITORING EVALUATION 



SOURCE 
ESSENCE 



Extension education. 



Monitoring evaluation deals with the aritioal management 
taeke in keeping programs on track. 



USE 



Particularily useful in programs you're doln^ for the first 
time or in programs that have fallen into a traditional 
routine. 



STARTING 
POINT 



Start with the program plan, statements of the expected 
outcomes, and standards for the processes to be used. 



DESCRIPTION Evaluation shouldn't be confined to the end of the 

program process, but should be used continuously. Monitoring 
is oontinuouB rather than episodic. Monitoring evaluation 
is like using a combination of a road map and observations 
of local landmarks in determining how far you are from your 
destination. 

The in5>ortance of using systematic monitoring depen4s 
on; (l) the program's responsiveness to direction and 
(2) the degree of risk associated with failure of the 
program. Which activities are irreversible and therefore 
need to be examined before they reach the point of no return? 

Major elements of the program used in the model are: time, 
program action^ and expected transitional outoomea. All 
programming occurs within time frameworks. Program action 
occurs over time. Expected transitional outcomes exist in 
time. Eocpeated traneitionat outoomea describe what must 
occur at various time slots so the ultimate outcomes will 
be attained. They're substeps distributed through time 
and become a source for criteria and standards for monitoring 
the program. 

Evaluation cycles should occur as part of each decision 
in the programming process. Evaluation cycles may also 
Increase the amount of decision making that's done. 

An evaluation cycle consists of four stages: 

1. D^eigTl— deciding on what criteria and standards to use 
to determine the success or progress to that point. 

2. Obaervatione-^^eciiing on what data to get. Data 
may come from direct viewing or from collected 
information* 
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3. Judgment — deciding what adjustment, if any, needs to be 
made in the program. 

/cyuatment— using the feedback in adjusting the 
program* 

The rapidity with which evaluation cycles will occur depends 
on the frequency with which decisions are required, the 
amount of activity in the prograiraning process, the number of 
subsets of transitional outcomes, and the degree of anxiety 
with respect to the outcome. 



Bruce, Robert L. "Monitoring Evaluation." Ithaca, Mew York: 
Cornell Ttiiversity, 1972. [Hlneographed!] 



4.5 IMPROVEMENT EVALUATION 



ORIGIN Indiana adult basic education* 

ESSENCE This appraisal procedure was developed at the request of 

state directors. They selected priority areas for examina- ' 
tion where they felt evaluation could improve programs. 

USE Excellent when improvement rather than proof of results is desired. 

STARTING Identify the purpose of the evaluation. 

POINT 

DESCRIPTION One state's program is seldom better or worse than other 

states. Each has its own unique problems. When the purpose 
of evaluation is improvement rather than external justifica- 
tion or an attempt to improve status, the approach can be 
more relaxed and less trauma -^producing at the local level. 
The judgment of outsiders can play an important catalytic 
role in helping local programs identify areas where improve- 
ments need to be made, identify alternatives in areas where 
improvements must be made, and reinforce confidence in .those 
areas where the program excells. 

The following process is indicated: 

1. Those commissioning the evaluation meet and define 
the specific purpoeee that the evaluation's ex- 
pected to achieve. 

2# Areas to be examined are identified and priovitiee 
set* The sponsoring group develops a list of 
aspects that might be examined. Priorities are 
established by each member indicating how he sees 
priority; then the various views are combined. In 
the Indiana example, the areas that were selected 
were: priority 1: curriculum, reading program, 
recruitment, teacher training; priority 2: administra- 
tive and supervisory control, administrative relation- 
ships, community relationships, counseling program, 
learning laboratories, placement, public relations, 
teacher use of human resources. Other areas that 
were suggested, but finally classified as non-priority 
were: mathematics program, relationship of local to 
state staff, staff relationships, teacher use of material 
resources, program objectives, student records, and 
teacher attitudes. 



3. Consultants are selected according to the expertise 
needed in the hicbest priority areas* 



Which local programs will be examined is determined 
through a sampling procedure. In the Indiana 
example, part of the sample was purposely stratified 
with the state staff selecting programs according 
to rural-urban location and geographic representation* 
An equal number were chosen by a table of random . 
numbers. More than half of the 3U local programs 
were included. 

5. Consultants use a set of general guidelines but 
are free to assess and comment on any aspect that 
impresses them either for its strength or weakness* 

6. Observations are made by visiting classes; observing 
students; talking with teachers; talking with students 
contacting community citizens; talking to dropouts; 
talking to superintendents, coiuiselors, supervisors > 
ABE directors, and relevant others. 

7. Consultants file written reports with case 
examples and recommendations. The reports are 
combined into an overall report that briefly discusses 
and gives recommendations on each of the priority 
areas and then summarizes the overall strength and 
areas where improvement is needed. 

8. The report becomes source material for state level 
programmers in developing in-service training and 
in making practical suggestions to local programs. 

The on-^ite consultant activity can be supplemented by: 

1. A study of needs in a rural and urban area. Inter-- 
views with potential clientele indicate the strength 
of need and some of the barriers to reaching the 
clientele. 

2. Use of previous reports and studies so that the new 
effort builds on what^s already known rather than 
repeating such efforts. 

3. Examining objectives of programs and/or alternative 
types of information. In the Indiana example, an 
earlier study had indicated that only part of the 
programs had clearly defined statements of objec- 
tives. Rather than asking for such statements, local 
programs were asked to provide one example of what 
they consiiered the b^st achievement of their program 
and one example of good individual gain through the 
program. It was assumed that these data would reveal 
the objectives of the program. 



Kreitlcwt Burton W. Indtcma Adult Basic Eduoation^-^An 
Intpravment Evaluation. Indianapolis, Indiana: State 
Department of Public Instruction » 1972. 
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4.6 TRANSACTIONAL EVALUATION 



ORIGIN 
ESSENCE 



Evaluation of local programs* 



Transactional evaluation considers the belief e and 
feelinga of people as they react to suggested changes* 



USE 



Particularly useful in helping program personnel accept 
new challenges. It can be of particular help when you're 
doing a "transplant operation," that is, introducing a 
change that was developed outside of the system that will 
be using it, or when outside groups are pressuring for 
changes and staff are slow or reluctant to make these 
changes • 



STARTING 
POINT 



Identify how people will react to a suggested change in 
programming • 



DESCRIPTION Transactional evaluation looks at the effects of 

changed programs on the changers themselves* Its focus 
is on evaluating program acceptance. It contributes to 
program improvement to the extent that it facilitates . 
needed changes. 

Transactional evaluation considers the beliefs and 
feelings of people. It requires that both the supporters 
and the critics of a program — ^those who will be favorably 
and those that will be adversely affected — be included in 
the planning and carrying otit of the evaluation, and in the 
forming of conclusions. It's concerned not only with the 
improvement of a given program, but with the program's 
effect on the system. In particular, it's concerned with 
dysfunctions that may occur. 

Attention is given to examining: 

1. Unanticipated as well as expected consequences. 

2. Effect on the total organization and not just those 
directly involved. 

3. Dislocation due to competition for resources (including 
student time and attention) and shifts in roles and 
expectations. 

This examination is seen as a continuous process in which 
the effects of the program are examined by both supporters- 
and critics* 
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Transactional evaluation differs from traditional summative 
and formative evaluations in that: 

1. The target of evaluation is different. The subject of 
the evaluation is the system, not the client or the 
services rendered* 

2. The variables relate to the social, psychological, 
and communications aspects of the system more than to 
the manifest objectives. 

3. Information would be continuously fed back into the 
system. 

4. The evaluator would be more a part of the operating 
system. 

5. Conventional considerations of reliability, validity, 
and objectivity would be less important than time- 
liness, relevance, and observable effects of the 
generation of evaluation information. 

6. The aim wouldn't be that of production of new 
knowledge or the attribution of causality, but , 
would be that of transforming the conflict energy 
associated with change into productive activity, 
and the clarification of the roles of all persons 
involved in changes in the program. 

Rippey, Robert M. "Can Evaluation Be Integral to Innovation?" 
Cuvnaulum Evatuatian: PotenHatity and Reality j Joel 
Weiss, ed. Curriculum Theory Network Monograph 
Supplement. Ontario, Canada: Ontario Institute for 
Studies in Education, 1972, pp. 

Rippey, Robert M, "Introduction: What Is Transactional 
Evaluation?" Paper presented at the meeting of the 
American Educational Research Association in Chicago, 
1972, 



GROUP 5 

Results—Attainment of Objectives 



OVERVIEW: 

Evaluation approaches that deal with examining results as specified 
in objectives are the most common and well developed of all evaluation 
models. Rather than including individual models in this section, approaches 
are grouped to indicate areas where development is going on. Distinction 
is made between approaches that are primarily concerned with the results of 
instruction (specific behavioral change) and those that focus on programs. 

EXAMPLES INCLUDED: 

0 The Tyler model and its adaptations. 

• Inatvuatiomt evaluation approaches. 

• Program objeoHvee approaohee. 

0 Objectives: oategorizationa <md criteria. 

GEHERAL ADVANTAGES: 

0 Using objectives to provide the major emphasis for examining 
results makes the whole evaluation process much more efficient . 

Putting some degree of emphasis on objectives also helps the program 
unit increase its skill in identifying what objectives are reason- 
able in what programming situations and develop its skill in using 
objectives as important tools in programming. 

• The question probably isn't one of whether objectives should be 
used in examining the results of programs. It probably should be 
whether any results other than those specified in the objectives 
should be examined. 

GENERAL CAUTIONS: 

, • Models that limit themselves to examining the extent to which 

1 program objectives have been attained often miss evidence of other 
j important results. Unexpected side effects can be either positive 
1 or negative f but often consideriaibly enhance or detract from the 
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overall success of the program. 
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All models that invest evaluation resources in relation to objectives 
assume (1) that the objectives themselves are well developed, 
realistic, and attainable; best suited to existing need?;; and the 
best choice among various objectives that could be attained; and 
(2) the design and implementation of the program is such that 
sufficient input and interaction occurred for the objectives to 
have actually been attained* Too often, however, one or both 
of those assumptions fail to be correct and any elaborate data- 
gathering procedure that focuses only on objectives produces 
little that^s of value to the program. 

Evaluation procedures that stress the whole group attaining the 
same objective may be inconsistent with the philosophy of 
education that places the emphasis on individual needs and individual 
differences. In the latter case, the objectives used should be tailor- 
made to and probably by the individual student. Then judgments of 
programs would be on the extent to which a program helped an 
individual achieve his objectives rather than only on the extent 
to which a group of people were brought to attain an objective 
set for the total group. 

Other limitations come not so much in the models themselves but 
in the intent in which they are used. Models often delete or* 
de-emphasize the steps in the original Tyler uiodel that relate 
to analyzing the strengths and weaknesses of the program by 
using data about the attainment of objectives. 
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5*1 TYLERIAN MODELS 



General education. 



Evaluation is concerned with determining whether education 
is actually producing the results that it sets out to achieve 
as indicated by statements of objectives. 

Can be used in any program that has objectives. However, 
it's most apt to be successful when there's evidence 
that the objectives are realistic and the program input 
great enough to make it logical to expect results. 

Analyze the objectives to determine exactly what results 
are expected. 

THE ORIGINAL The role of evaluation is seen 

TYLER MODEL as improving curriculiim by finding 

out how far the learning experiences 
as developed or organized are actually producing the desired 
results; identifying strengths and weaknesses of teaching 
plans • 

The evaluation process includes: 

1. Analyzing objectives to identify and clarify their 
two basic dimensions* Those dimensions are: 

Ca) the behavior expected from the student and 
(b) the key content that is to be mastered. 

2. Identifying situations that will give the student 
a chance to express the behavior related to the 
content • 

3» Selecting or developing instruments that will 
record the behaviors. 

Analy?:ing the amount of change that's taking 
place in studcVcto* 

5. Analyzing the strengths and weaknesses of the 

program in helping students achieve th(> objectives* 

Prerequisite to this process, objectives have been screened 
against the heeds indicated by various sources~the student, 
society, and the discipline-*^nd the screens of philosophy 
and learning theory to be dyxre that they're appropriate. The 
learning experiences Included have also been evaluated against 
criteria* Ibis type of evaluation t then, focuses on the 
behavior of students* 



Evaluation must appraise the behavior of students. This 
appraisal must occur more than once to see whether a change 
has occurred. Any valid evidence about the specific behavior 
provides an appropriate method of evaluation. Sanipling is 
important in many ways including a sample of the student *s 
work. Several pieces of evidence will be used to form 
profiles rather than relying on one piece of evidence. 

Curriculum planning is a continuous process. As materials 
and procedures are developed, they* re tried out, results 
appraised, inadequacies identified, and suggested improvements 
indicated. Replanning, redevelopment, and reappraisal are 
a continuing cycle. 

SOURCE: Tyler, Ralph W. Baaio RHnciplee of Currioulm 
ccnd Inetruotion* Chicago: University of 
Chicago Press, 1950. 

NATIONAL Although developed for specific 

ASSESSi.JNT PROGRAM curriculLon, the Tyler framework has 

recently been used for a national 
assessment. The plan for the National Assessment Program 
developed by the Exploratory Committee on Assessing the Progress 
of Education made the following decisions that carried ,out 
the basic model: 

!• Focus would ba on age levels. Four were chosen. 

2. There would be no attempt to evaluate individuals 

or specific school systems or programs. Sample would 
be of the whole age population with subsampling 
stratification based on sex, region of the country, 
type of community, =ind socioeconomic background, 
but not school systems. 

3. £ducational objectives determined what data as to 
educational results would be collected. Objectives 

. selected were those that: (a) school systems said 
they were reaching, (b) scholars considered authentic 
in the various disciplines, and (c) discerning lay 
ciults thought were significant • Considerable 
negotiation was requii^ed to get agreement within 
and among the three groups. 

For each objective there was an attempt to describes 
(a) things that almost all persons of the age level 
have accomp^lished, (b) things that average persons 
have achieved, and (e) things the most advanced at 
that age level could do. 

5« The data-fathering exercises were packages so that 
differenc students did different things* For example. 
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30 students spending 40 minutes taking different sets 
of exercises would provide 1 response to each of a 
total set of exercises that would take 1 student 
20 hours to complete. With 15,000 students responding 
to the various sets, there would be 500 responses for 
each set of exercises. 

SOURCE: Merwin, Jack C, and Frank B. Womer. "Evaluation 

in Assessing the Progress of Education To Provide 
Bases of Public Understanding and Public Policy." In 
Edmational Evaluation: Hew Rotee^ New Meane^ 
Ralph Tyler, ed. Chicago: The University of 
Chicago, 1970, pp. 306-12. 

FOUR-QUESTION Gottman and Clasen define evaluation 

APPROACH as quality aontrol of the processes 

and outcomes of an educational program. 

They built an evaluation text around four Tyler questions: 

1. Wij/?— What needs can you cite that justify the existence 
of this educational program? 

2. \ifhat? — What are your objectives in the program? . That 
is, what objectives will the program .iccomplish to 
meet the need under why? 

3. Raw? — How will you have the program "unction to meet 
its objectives? 

4. Row Will lou i&KTW?— What kinds of information should 
be gathered so that you know if the how id meeting 
the wiuit for the why? 

They suggest the following activities in relation to the 
questions: 

Why? Needs assessment. 

What? Writing measurable objectives and 

designing measurement procedures. 

Row? Flowchart . 

Row Will lou Know? Design and use quality contiK>l pro- 

cedures . 

Emphasis: time-series analysis 
procedures. 

SOURCE: Gottman, John Nordechai» and Robert Earl Clasen. 

Er)aluabion in Edmaticn: A PraotioMv^e Ouide. 

Itasca, ininols: F* E. Peacock Publishers, Inc., 1972. 
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ADOPTION OF THE During the X950s and 1960s, adult 

TYLER MODEL BY education including Extension, 

ADULT EDUCATION pretty well adopted the approach 

presented in the Tyler model. 

Although it usually was not called program evaluation, it 

served in that capacity. 

In 1952, the Committee on Evaluation of the Adult 
Education Association formulated this definition; "Evalua- 
tion is the process of assessing the degree to which one is 
achieving his objectives. It is looking at one's present 
position in regard to one's goal. It is a comparison of 
the actual with the ideal." The committee listed the 
following principles: 

1. Self -appraisal usually is better than appraisal 
by outsiders. 

2. Comparison of achievement witL objectives within 
a program leads to more growth than comparison of 
one program with others. 

3» Everyone concerned with the educative process should 
be involved in evaluation. 

Evaliiation offers greatest potential benefit if it's c 
long-^time, continuous, and built-in part of the total 
educational process. 



5. Evaluation should be more concerned with results than 
with energy spent. 

6. Objectives should recede as they're approached. ; 

In the chapter on evaluation in a book on outlines of 
an emerging field of university study, Theide defined ^ 
evaluation as the process of determining the extent to which 
educational objectives have been attained. He listed 
five steps in the evaluating process: (1) determining 

what to evaluate, (2) defining the behavior desired, 1 
(3) determining acceptable evidence, (^) collecting { 
evidence, and (5) summarizing and evaluating the evidence. ; 

Objectives were seen as being essential and derived 

from values and needs of society, purposes of institutions, * 

needs and inte^'ests of individuals, the subject matter it- - 
self, and learning theory. Objectives snould be achievable, 

should be in harmony with other objectives of the educator, i 

should be able to be built on for further objectives, should J 

be agreed on and have common meaning, end should be closely | 

related to desired learner behavior. Objectives need to I 

be expressed in specific behaviors desired in the learner. i 
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Situations must be contrived that will permit the learner to 
exhibit the desired behavior. These situations must be 
appropriate and relevant, must control unwanted factors, 
and must be practical and reliable. 

Records of behavior can be made by the learner himself, by 
another observer, or by mechanical devices and should be 
objective, reliable, and valid. When they've been summarized 
by counting, describing, and analyzing, judgments ar^ made 
to show the extent to which objectives have been attained, 
appropriateness of the objectives themselves, and the 
effectiveness of the learning experiences. Evaluation must 
always suit the purposes to be achieved and it must be 
practical. 

The adult learner is task oriented, has specific objectives 
of his own in mind, and is less willing to accept unknown 
objectives or the objectives of others. He, therefore, 
needs to be involved in the establishing and accepting of 
objectives in the learning situation. Self -evaluation needs 
to be developed and experimented with in adult education . 
and incorporated in the learning process. Variants in 
adult education programs create special problems. The tendency 
for programs to be informal, of short duration, and baaed 
heavily on learner objectives makes evaluation by outsiders 
less acceptable and makes measurement itself more difficult 
because of the coarseness of available devices and techniques. 

Miller and McGuire used the Tyler model as a basis for 
studying methods for evaluating liberal adult education4 
They used three criteria for the instruments that they 
developed. 

Reatiem: The closer we can come to the realities 

of adult life in our testing, the better 
able we will be to claim that we've changed 
students' habitual cognitive behavior. But 
* more than that, tests that are clearly related 

to real-life problems will also be more 
interesting and reduce resistance to the test- 
ing situation. 

Intereet: Every possible f,ay of making the instruments 

themselves interesting to work on will increase 
the chances of their being used. They should 
challenge and Involve the person who takes 
them. 

FUffibitity: The several evaluation devices should be useful 
to the widest variety of educational programs, 
and therefore need to be developed in such a way 
— " that any individual program may select from them 
those parts that are particularly relevant. 



159 



A manual produced by the Extension Research and Training 
Division for Cooperative Extension Employees. guided 
Extension in combining a Tyler approach with research-type 
efforts for examining results. 

In 1966, Sabrosky wrote the chapter on evalu^rior. for 
the new book on the Cooperative Extension Servxc^. She 
quoted the Tyler definition of evaluation, but discussed 
briefly the evaluation of Oi'ganization objectives n^id 
evaluation of personnel before examining the place of 
evaluation in £xtension program development. She outlined 
a series of questions to be examined in evaluation during 
the planning stage and a parallel su«: to be examined in 
terms of the resultant program: 

In planning for Extension education, the situation is 
evaluated. In doing so, these questions are asked: 

1. Who should be helped to move toward an objective? 

a. What are these people like? 

b. What problems do they have? 

c. Which information and skills do they already have? 

2. Which problems should these people get help in solving? 

3. Which objectives will be the best ones at which to 
aim? 

4. What resources do they have? 

5. What are their motivations? 
These questions are then asked: 

1. How should these people be helped to move toward each 
. objective? 

a. Which methods will be most appropriate? 

b. Which subject matter will be most appropriate? 

c. When should the teaching be carried out? 

d. Who should do the teaching? 

In evaluating Extension education, processes, and results, 
these questions are asked: 
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1. were helped to move toward an objective? 
a, Whal: were they like? 

bv What problems did they have? 

c. Hlict did they learn? (Behavioral changes) 

2. Did these behavior changes seem to help improve the 
situation? 

3. What resources did they lack? 
What were their motivations? 

These questions were then asked: 

1. How were the people helped to move toward an objective? 

a. Which teaching methods were most effective? Why? 
With whom? When? By whom? Least effective? 
Why? With whom? When? By whom? 

b. Which subject matter was most useful? Why? . 
With whom? When? By -whom? 

c. Wat changes are implied for future Extension programs? 

In 1970, Boyle and Jahns defined program evaluation as 
the determination of the extent to which the desired objectives 
have been attained or the amount of movement that has been 
made in the desired direction • They saw it as being primarily 
concerned with assessing the ends that were attained and not 
div>ectly concerned with assessments about other aspects of 
the program such as the means used to attain objectives. Seen 
in this light, effective evaluation requires: (l) clear, 
concise objectives or statements of intended, educational 
ends to be attained; (2) bench mark or pre-program measures 
of the behavior(s) or behavioral patte>»ns of the learner 
before his exposure to the educational program; and (3) measures 
after completion of the educational program, 

Sanders (1972) indicates that program evaluation could be 
conducted for any one or a combination of the following purposes: 

• To determine the situation. 

. 0 To determine the quality of performance. 

• To determine the effectiveness of learning experiences. 

• To determine the extent to which objectives are attained. 
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The existing situation is compared with what the situation 
should be. What was done is compared with what should have 
been done. Effectiveness of learning experiences is examined 
by comparing actual influence with potential influence or one 
experience with another for relative effectiveness. In deter- 
mining the extent to which objectives are attained, the behavior 
expected is compared with the behavior accomplished. 

Among other recent adult education texts, Houle (1972) 
holds closely to evaluation being the determination of the 
extent to which the educational objective has been accomplished. 
Axford (1969) focuses on three principles and examples of 
how to operationalize them. They are: (1) instruments for 
self -evaluation are preferred to evaluation by others, (2) in- 
volving program planners in the evaluation process is necessary, 
and (3) evaluation should be concerned with outcomes and 
results rather than activity or energy spent. Klevins chose 
only to include a chapter of self-evaluation which deals 
primarily with teacher performance in his 1972 book- 

Cass (1971) broke from the prevailing pattern and siiggests 
that evaluation of adult basic education should start with 
a study of the target population and the number of students 
enrolled. Other means of evaluating programs include: . number 
of dropouts, number completing each achievement level, nuinber 
going on to a higher level, persistence of attendance, 
estimation by teacher and guidance person of student's 
interest, number enrolling in other schools, follow-up on 
employment records. Evaluation should be continuous and 
include information about how adequately each student is 
attaining his goal(s), which methods are most effective, 
what content is most satisfying; which materials are most 
useful, what else teachers need to know. 

There's evidence that adult and extension education is 
using .newer models developed in education: for example, 
Wedemeyer, 1969; Raudabaugh, 1970; Farmer et al. , 1970; 
Hale, 1970; Elliott, 1972. And, as work in this monograph 
attests, some adult educators are striking out and developing 
models that are better adapted to adult education. Many 
of these models are coming from experiences in evaluating 
programs for disadvantaged adults and evaluating other 
specially funded federal progrc«ns. _ 

Knowles (1970) suggests that evaluation is a much over- 
emphasized sacred cow and that there 'r a conflict in values 
going on. At one end of the spectrum is the emphasis on 
"hard data" and values of such groups as experimental researchers 
and cost accountants. At the other end of the spectrum is 
an emphasis on self -actualization, artistic intuition, free 
play of natural forces » and creative ambiguity. An adult 
educator will choose the kind of evaluation that fits his phlloso] 



162 



As adult and extension educators move beyond the concept of 
program evaluation held in the 1960s, some may want to return 
to the older literature of adult education. For exaii5)le, 
Kempfer suggests a framework for examining the program in 
terms of its comprehensiveness in comparison with the needs 
of the community and suggest such questions as: What difference 
does the program make in the life of the community? Are there 
fewer broken homes? Less discrimination? Are adults be- 
coming more competent in solving both their individual and 
their group problems? Essert suggests criteria for desirable 
self-growth as appropriate in evaluating the effectiveness of 
adult education programs. Categories include: experience of 
occupationcd achievement, experience of understanding, 
experience of self-government, experience of close fellowship, 
experience of inter*mittent solitude. 

SOURCES; Committee on Evaluation. Progrm Evaluation in 
Adult Education. Wcishington, D. C. : Adult 
Education Association of the U.S.A., 1952. 

Tbiede, Wilson. ";3valuation and Adult Education." 
In Adult Education: Outlines of an Emerging 
Field of University Study^ Gale Jensen, A. A. 
Liveright, and Wilbur Hallenbeck, eds. Washington, 
D.C.: Adult Education Association of the U.S.A., 
196^. 

Miller, Harry L., and Christine H. McGuire. 
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Center for the Study of Liberal Education 
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Topeka, Kanscis: H. M. Ives 6 Sons, Inc., 1959. 

Sabrosky, LatD?el. "Evaluation." In The Ccoperative 
Extension Service^ H. C. Saunders et al . Englewood 
Cliflfs, New Jersey: Prentice-Hall, Inc. 1966, 
pp. 339-51. 

Boyle, Patrick G., and Irwin R. Jahns. "Program 
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^dult Educations Robert M. Smith, George F. Aker, 
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5.2 INSTRUCTIONAL EVALUATION APPROACHES 



ORIGIN 
ESSENCE 



General education. 



Precision in stating objectives facilitates instruction 
and evaluation. 



USE 



Particularly useful in evaluating changes in knowledge. 



STARTING 
POINT 



Specif objectives in very precise terms. 



DESCRIPTION Much of the current work on objectives and many of the 

recent interpretations of "educational" evaluation are 
specific to the evaluation of instruction rather than 
sufficiently broad as to encompass program evaluation. 

However, examining some of the more recent developments 
in instructional evaluation is useful in that some programs 
are made up primarily of instructional type activities (that 
is, classes or courses) and some of the basic ideas in 
instructional evaluation, for example, setting standards 
of performance for objectives can be adapted to program 
evaluation . 



GOAL-REFERENCED 
INSTRUCTIOU 



Popham and Baker have succeeded in 
translating a fairly complex 
curriculum procedure into a model 
acceptable to the average teacher. They suggest that a 
goal-<^ferenced instructional model has four main parts: 

1. Specification of objectives. 

2 . Preassessment . 

3. Instruction. 
^. Evaluation. 

Objectives are to be stated in measurable learner behavior. 
Two kinds of techniques can be used to judge the degree to 
which the learner has attained the .instructional objective: 
(1) observation of behavior and (2 examination of products 
that the learner will produce. For example, observing the 
student. giving a speech (behavior) or examining an essay or 
test that he has written (product). 

There are two dimensions to setting levels of performance 
in regard to an objective • One is the level of performance 
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for each student , the other is the level of performance 
for a group of students. Examples of such standards are; 
individual student, each student should attain scores 
of at least 90; group of students, 90 percent of the 
class should attain a score of 90 or better. Standards 
should be set for the minimal acceptable attainment. 

Objectives need to be sequenced for effective instruction. 
This sequencing helps in later decisions about how attain- 
ment will be evaluated. The following guides have been 
suggested: 

1. Analyze objectives into content-behavior modules, 
not just content. 

2. Use guidance from a regularized scheme such as that 
advocated by Gagn/or Bloom. 

3. To arrive at a first sequence, repeatedly ask the 
question: "What does the learner need to be able 
to do before he can perform this task?" 

Clearly identify prerequisites not within your 
instructional responsibility. 

6. Where possible, empirically verify sequences generated. 

After tentative objectives have been set, the teacher 
should use preassessment procedures to determine how many of 
those objectives are presently attained by the majority of 
the students and where further emphasis is needed by the majority 
and remedial work by a minority. Preassessment helps prevent 
reteaching when such teaching isn't necessary and helps identi^ 
where present emphasis should be placed. It also tests assump- 
tions about the kinds of prerequisites students will be bring- 
ing to the learning situation. 

The preassessment helps identify whether differentiated 
instruction is necessary. Such instruction provides two or 
more tracts suited to the general abilities within the group. 
Two of the most common ways of differentiating are: (1) setting 
additional objectives for more advanced students or (2) identi:fy- 
ing more challenging means for accomplishing the same set of 
objectives. 

Evaluation at the end of the teaching sequence consists of 
securing evidence either of the behavior of students or of 
examining products of that behavior. This evidence can be 
secured either under natural conditions or under manipulated 
conditions. Listening to grammar used at break time would 
be an example of securing evidence under natural conditions • 
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Tests are usually conducted under manipulated conditions 
in that the teacher deliberately sets up the stimulus 
condition. Tests are of two kinds. Some are norm 
referenced. Others are criterion referenced. Norm- 
referenced tests are those where irdividual scores are 
interpreted in relationship to the scores of others. 
Criterion-referenced tests compare performance to a set 
criterion without regard to the scores of others. Criterion- 
referenced evaluation of student performance differs from 
the usual grading discrimination system. A criterion- 
referenced approach to examining the performance of adult 
students may be more appropriate than is the traditional 
procedure of comparison with the work of others. ' 

In a guidebook developed under contract to the Bureau 
of Indian Affairs, Popham defines evaluation as the act 
of aeeessing merit by judgmentally comparing (1) the observed 
result (performance data) of some educational enterprise 
with (2) a desired standard or criterion of acceptability 
(preference data) . 

He suggests two major roles for this type of evaluation: 

Role One—Educational Needs Assessment. Determining the 
desired ends of the educational system. 

Role Two—Treatment Adequacy Assessment. Judging the 
worth of educational meane. 

Formative. To improve the treatment as it*s being 
developed. 

Suntmtive. To compare the treatment with competitors. 

The following guidelines for educational evaluators 
are presented after a discussion of the key ideas involved 
in the guideline: 

1. The educational evaluator should encourage the use 
of instructional objectives that provide explicit 
descriptions of the post-instructional behavior of 
learners . 

2. While recognizing that nonmeasurable goals will be of 
limited use for his purposes , the educational evaluator 
must be aware that instructors may wish to devote a 
reasonable proportion of their efforts to the pursuits 
of important but currently unassessable objectives. 

3. The educational evaluator must identify criteria of 
adequacy when: using instructional objectives that 
require constructed responses from learners* 



167 



The educational evaluator should foster the use 
of measurable objectives that possess content 
generality rather than test-item equivalence. 

5. Before the introduction of the instructional 

, treatment, educational evaluators should strive to 
establish minimal proficiency levels for instructional 
objectives . 

6. The educational evaluator will often find the taxonomies 
of educational objectives useful both in describing 
instructional objectives under consideration and in 

I generating new objectives. 

7. The educational evaluator should consider the possibility 
of selecting measurable objectives from extant collec- 
tions of such objectives. 

8. The educational evaluator should avoid the use of 
norm-referenced measures, preferring instead 
criterion-referenced measuring devices . 

9. The educational evaluator should use domain-referenced 
achievement testing procedures when called on to 

j generate test items for objectives or to determine 

! the content validity of already developed test items* 

10. The educational evaluator should encourage the use of 
multiple criterion measures by generating alternative 
schemes for assessing learner behaviors. 

11. When the use of customary measuring tactics may produce 
reactive effects, the educational evaluator should 
employ unobtrusive measures. 

12. The educational evaluator should be particularly 
attentive to. unanticipated outcomes that result from 
an instructional treatment, and should use such 
outcomes in judging the merits of the treatment* 

13. The educational evaluator should clarify value 
preferences of various groups regarding desired 
educational goals by having sets of precisely 
stated objectives rated by the individuals involved, 
then translating these ratings into composite 
indicators of each objective's worth. 

14. The educational evaluator can compare preference and per-* 
formance data by contrasting preference x-atings of 

I - objectives with measures designed to assess the degree 

to which learners have achieved those objectives* 
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15. Whenever large scale measurement is required • the 
educational evaluator should consider the ecc lomic 
advantage of employing item-sampling and person- 
sampling procedures. 

16. In conducting formative evaluations t the educational 
evaluator will find it useful to (a) employ small 
samples of learners, (b) secure performance data 
about terminal and en route objectives, and (c) use 
designs such as the one-^hot case study or the one 
group pre-test/post-test design. 

17. In conducting summative evaluations, L'he educational 
evaluator should prefer the use of designs invo3.ving 
randomized control groups, that is, the pre-test/post-* 
test control group design or th6 post-test only 
control group design, but when randomization is 
impossible can profitably employ the nonequivalent 
control group design or the interrupted time series 
design. 

18. The educational evaluator shotild analyze data according 
to the smallest independenL units available, frequently 
leading to the use of classroom or larger units .rather 
than individual pupil units. 

19. The educational evaluator should, in general, prefer 
descriptive statistics and estimation procedures 
instead of statistical hypothesis testing procedures. 

20. The educational evaluator should present decision makers 
with a wide range of pertinent information so that 
choices among alternatives can be made in a cost^ffective- 
ness context* 

/ 

SOURCES: Popham, W. James, and Eva L. Baker. Syetematvo 
Instruotiona. Englewood Cliffs, New Jersey: 
Prentice-Hall, Inc., 1971. 

Popham, W. James. An Evaluation Guidebook. Los 
Angeles; The Instructional Objectives Exchange, 
1972. 

k 

EVALUATION AS 
FACILITATION OF 
LEARNING 

'^^^ and iin>roving learning 

as encompassing: 

1. Evaluation as a method of acquiring and processing 
the evidence needed to Improve the student's learning 
and the teaching. 



A 1971 handbook by Bloom, Hastings, 
and Madaus is concerned with putting 
the evaluation of Instruction and the 
evaluation of learning back together 
as a result. The handbook is viewed 
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2, Evaluation as including a great variety of evidence 
beyond the usual final paper and pencil examination, 

3, Evaluation as an aid in. clari^ing the significant 
goals and objectives of education and as a process 
for determining the extent to which students are 
developing in these desired ways. 

^. Evaluation as a tool in education practice for ascertain^ 
ing whether alternative procedures are equally effective 
in achieving a set of educational ends. 

The authors present a two-dimensional figure showing the 
relationships among analysis of the learners, instructional 
decisions and evaluation (row headings); and student, 
insti?uctional process, and objectives (column headings). 

The three stages of evaluation that correspond to the coluonn 
headings and some of their activities a2*e: 

Initial Evaluation 

1» Record of previous relevant achievement. 

2. Diagnostic and placement tests. 

3. Aptitude tests relevant to the choice of alternative 
learning approach .os • 

Formative Evaluation 

1. Tests relevant to the structure of eacn learning 
unit or task. 

2. Analysis and diagnosis of what must still be learned. 

3* Prescription of alternative learning materials 

and approaches needed to master the unit or tasks. 

S umniative Evaluation 

1. tx sample of the attainment of objectives and content 
in the outcome model. 

2. Analysis and diagnosis of extent w>^ich various 
parts of the outcome model have been attained. 

3# Analysis and diagnosis of extent to which each 
learner has attained the outcomes on the table of 
specxficatlons. 
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The outcome model or table of specifications is a grid 
that lists the key elements of content of the course or 
unit on one dimension and the expected behaviors related 
to that content on the other dimension. Cells are then 
formed coupling behaviors with content to indicate potential 
objectives. The cells actually used to frame the expected 
outcomes are judged in terms of whether they are both 
possible and desiruble 

SOURCE: Ploom, Benjamin S., J. Thomas Hastings, and George F. 
Madaus. Handbook on Formative and Sumative 
Evaluation of Student Learning. New York: 
McGraw-Hill Book Company, 1971. 



Blaney, John P. "The Application of Formative and Summative 
Evaluation Techniques to Short Terra Adult Education 
Courses." Paper presented at the Adult Education Research 
Conference, Chicago, Illinois, 1972. 

Popham, W. James, Elliot W. Eisner, Howard J* Sullivan, and 
Louise L. Tyler. Inetruationat Objeotivee, AERA Monograph 
Series pn Curriculum Evaluation, No. 3. Chicago: Rand 
McNally |6 Company, 1969. 

Evaluation Corment. Los Angeles: University of California, 
Center for the Study of Evaluarion. ' [Periodical] 

Wilhelms, Fred T*, ed, Evaluavion Ae Feedback and Guide^ 

Washington, D.C: Association for Supervision and Curriculum 
Development, NEA, 1967. 

irtttrockj M. and D» Riley. Evaluation of Inetruotion: leeuee 
and Probtem. New York s Holt,. Rlnehart and 
Winston, 1970. 
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5.3 PROGRAM OBJECTIVES APPROACHES 



ORIGIN 
ESSENCE 



Education and other social programs. 



This group of approaches combines objectives with other 
elements such as criteria or program components. 



USE 



Helpful in getting a broader understanding of how programs 
achieve objectives • 



STARTING 
POINT 



Define the objectives and identify other relevant components, 



DESCRIPTION This set of evaluation frameworks either specifically 

says it*s concerned with objectives that go beyond the 
kind of objectives that ax^e found within ^ specific course 
or instructional unit, or is so phrased that it can be 
applied in a broader setting. 



CRITERIA OF 

SUCCESS 

APPROACH 



"Evaluation is the d< termination 
(whether based on opinions, records, 
subjective or objective data) pf the . 
results (whether desirable or» un- 
desirable; transient or permanent; immediate or delayed) 
attained by some activity (whether- a program or part of a 
program ... an ongoing or one--shot approach) designed 
to accomplish some valued goal or objective (whether ultimate, 
intermediate, or :.amediate, effort or performance, long or 
short range). This definition contains four key dim^=»nsions : 
(1) process-^-thc *det elimination,* (2) criteria — *the results,' 
(3) stimulus— the 'activity,' and (4) value— the 'objective.' 
The scientific method with its accompanying research techniques 
then provides the most promising means for 'determining' the 
relationship of the 'stimulus' to the 'objective' in 
measureable ' criteria ' . " 

Five general categories of criteria are suggested against 
which the success or failure of a program may be evaluated. 
Attention is given to a sequence of objectives from immediate 
to intermediate to long range. An understanding of the 
assumptions that underlie the objectives and a checking 
of the validity of those assumptions is essential. 

Five general categories of criterie. according to which the 
success or failure of a program in attaining its imraediate-, 
intermediate-, or long-range objectiver may be evaluated are: 
effort^ perfomcmoe^ adequaoy of perfomanoes effioienoys 
and ptK>ceee^ 
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Effort. This category includes criteria concerned with 
the quantity and quality of activity that takes place. 
It deals with questions like: What was done? How well 
was it done? This criterion area can be compared with 
measuring the number of times a bird flaps its wings 
without trying to determine how far it flies. 

Performance. Criteria in this category are concerned 
with the result of effort. Questions include: How 
much is accomplished in terms of the immediate goals? 
Did any change occur? Was the change the intended one? 
Using this type of criteria is comparable to examining 
how far the bird has flown instead of merely counting 
the flappings of its wings. 

Adequacy of Performance. This category of success 
criteria deals with degree to which the effective performance 
is adequate to the total need. How many of the people 
who are potential users of the program actually are reached 
and actually use it? How uuch impact has the program had on 
the needs and problems that initiated it? Judgments based 
on criteria in this category must also be conscious of 
reality in terms of what can be expected with a given level 
of knowledge and available resources. In terms of the. 
analogy of the bird, this category is concerned with how 
far the bird has flown compared to how far it needs to go. 

Efficiency. Efficiency is concerned with the evaluation ^ 
of alternative paths or methods iv 'i:erms of costs — money, 
time, personnel, and public convenience. It represents 
a ratio between effort and performance and deals with ^uch 
questions as: Is there any i^etter way to attain the same 
results? In the bird analogy, comparable questions would 
be: Could the bird have arrived at his destination more 
efficiently by some other weans than flying the way he did? 
Did he take advantage of air currents? Did he fly too high 
or not high enough? 

Process. The process dimension isnH parallel with the 
other four categories, but is added because it helps make 
sense out of evaluation findings* It helps locate weaknesses 
anu identify whether the program can be improved or whether 
it should be dropped. 

Four dimensions of process are suggested: (1) the 
attributee of the program itself, (2) the population 
exposed to the program, (3) the eituational context 
in which the progra^i takes place, and (k) the different 
kinds of effects produced by the pwgram. Within the 
attributes of the program » we search the component parts 
of the 4 ^ogram for the one or ones that most contribute to 
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or detract from success. The population dimension explores 
such questions as: Which people are most affected by the 
program? Who do you succeed in reaching and who don*t you 
reach? Who makes the best target population for a program — the 
individual, ''he group, the public? Do you reach them as 
ultimate tai^-t? The third dimension explores whether the 
conditions greatly affect the success of the program. Could 
the same program be set up under different conditions and be 
as successful? 

The fourth category, effects,- can be broken down In various 
ways: unitary or multiple effects, unintentional or side 
effects, duration of effects, and type of effect. In each case, 
being able to specify the components existing in the actual 
program situation helps you better understand the success or 
failure of the program. 

SOURCE: Suchman, Edward A. Evaluative Reaearah. 

New York: Ri!ssell Sage Foundation, 1967. 

O-A-E Program performance is evaluated 

MODEL by examining the effectiveness and 

efficiency of the program. Evalua-- 
tion of both characteristics focus on the relationship of three 
key components: objeativeay aativitiee^ and reeouroee. These 
three components are defined as follows. 

Objectives. A situation or condition of people or of 
the environment that responsible program personnel corsider ' 
desirable to attain. (Objectives themselves include 
ultimate objectives, program objectives, and subobjectives. ) 

Activity. Work performed by program personnel and 
equipment in the service of an objective. 

Resources. Personnel, funds, materials, and facilities 
available to support the performance of activity. The followir.g 
abbreviations are uscd in the model: 

AO Attainment of objectives that can be attributed to 
the program activity. 

PO Proposed objectives for attainment through the 
program activity. 

AA Actual activities performed. 

PA Planned activities perfonned. 

AR Actual resourca expenditure. 

PR Planned resource expenditure. 



Effectiveness, in general, questions concerning 
effectiveness are directed toward assessing the extent 
to which a planned or intended objective has been attained 
as a result of program activity. An analysis is thus 
suggested in which the proportion of attainment of program 
objectives that's attributable to program activity (AO) is 
compared with the desired level, which during the planning 
process, the planners had proposed would result from the 
program activity (PO). Program effectiveness is denoted 
as the ratio, AO:PO. However, two subordinate measures 
must be considered to determine the soundness of the 
assumptions. One is the extent to which the activity has 
been carried out and has used resources in the way planned. 
In other words, the ratio of actual activities to planned 
activities, AA:PA. The second is the ratio of the actual 
expenditure of resources to the planned expenditure, AR:Pa. 

Efficiency. Efficiency is the ratio between an output 
and an input, AO:AR. The reverse, AR:AO, yields^.a measure 
of average cost. These measures of overall efficiency may 
be interpreted by examining two inteimcdiate efficiency 
measures: activities performed to resources expended, 
AA:AR or AR:AA; an'J objectives attained to activities 
performed, AO:AA or AA:AO. 

Continuous evaluation measures can be obtained in the 
following sequence:' 

1. The extent to which resources are being expended 
as planned, AR:PR. 

2. The extent to which activities cire being performed 
in the quantity and quality planned, AA:PA and 
the efficiency of resource expenditures, AA:AR. 

3. The net attainment of selected subobjectives, 
AOsub:POsub and efficiency of subobjective attain- 
ment, Osub:Rsub and Osub:Asub. 

^. Program effectiveness, AO:PO; program efficiency, 
AO:AR:; and activity efficiency, AO:AA. 

SOURCE: Denis ton, 0. L., e t al. ''Evaluation of Program 
Efficiency." Public Health Reports, LXXXIII 
CNo. 7, 1968). 

CRITERIA AND This outline for the evaluation process 

OBJECTIVES looks as though it should have been 

included in the section on program 

models. However » the {recess is applied primarily to the 

purpose and objectives of the prograa. 
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!• ?urpo8e of Evaluation. 

a« Improving a program or specific component of a 
program. 

. b. Proving value and importance of a program* 

2. Criteria Seleotian. 

The criteria used will depend largely on the program 
characteristics being evaluated. The program characteris- 
tics most frequently evaluated are: 

a. Effectiveness . 
b» Efficiency. 

c. Suitability. 

d. Importance. 

3. Preparation for Evidence Colteation. 

a. Determine program characteristic(s) to be evaluated 
(mentioned above). 

b. Determ±ne*"ifhat program level or component is to 
be evaluated. 

c. Determine program stage(s) to be evaluated. 

1) Program determination. 

2) Program preparation. 

3) Program implementation. 

Program evaluation. . * 

4. Evidence Colleotion. 

a. Sources of evidence. 

b. Evidence collecting methods and devices, 
t:. When to collect the information. 

d. Who should be involved in collecting evidence. 

5. Analyeia and Interpretation. 

a. Organizing and clrssii^ing evaluation data. 
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b* Making comparisons and judgments. 
6. Reporting and Uee^ 

a. Identify audience(s)* 

b. Identify how the report will be used. 

c. Report format. 

Criteria are derived from the objectives. Generally 
it's more difficult to develop criteria for high level 
(more general) objectives such as those that describe the 
purpose of a program than it is to develop them for 
specific instructional objectives. 

Efficiency is discussed in terms of alternative methods 
for securing the same objective. Suitability deals with 
the appropriateness of the objectives. Importance deals 
with the overall value of what's accomplished through 
attaining these particular objectives. 

SOURCE: Matteson> Harold R. The Proceaa of Ivalmting 

Vocational Eduaational Program in Agriculture. 
Instructional Series No. 3. Madison » Wisconsin: 
University of Wisconsin* Deportment of Agricultural 
and Extensi^on Education, February, 1972. 

LOCALLY DIRECTED Byram and Robertson present a framework 
EVALUATION that starts with objectives and gathers 

evidence about how well they've been 
attained. Emphasis should be on the program objectives and 
philosophy of education • Objectives of cour-^e and instruction- 
al units contribute to but don't constitute program objectives. 
The next steps deal with formulating criterion questions; 
identifying and obtaining evidence; analyzing, interpreting, 
and reporting information; and formulating and implementing 
recommendations. The process includes examination of input, 
process 5 and community needs. 

The unique contribution of this particular model is that it 
recommends that the evaluation involve local people (staff and 
citizens) in making the evaluation. They list the following 
essential elements in local evaluation: 

1. A commitment to the evaluation effort by the local 
administration • 

2. A strategically placed local leadership team. 

3. An active staff steering committee. 
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^. A functioning citizens* committee, advisory to the 
staff. 

5. An allocation of time to the leadership team. 

6. Training in research and evaluation procedures. 

7. Communications and visibility. 

8. Qualified consultant service. 

SOURCE: Byram, Harold M. , and Marvin Robertson. Locally 

Directed Evaluation of Local Vocational Education 
Programe. East Lansing, Michigan: Michigan 
State University, College of Education, Depart- 
ment of Secondary Education and Curriculum, 
March, 1970. 

OHIO Most attainment of objectives models 

MODEL are applied to specific local programs. 

However, models have been developed to 
apply the approach at a state or national level. 

Ohio reviewed existing methodological approaches to .the 
evaluation of vocational education and formulaf^d a system 
that included: 

1. Program objectives and measurable goal statements. 

2. A data set, instrumentation, and procedures to 
measure the effectiveness of efforts to achieve 
specific program goals. 

3. The means by which the evaluation system would be 
articulated with larger- management systems of state 
divisions of vocational education. 

A prerequisite was that the system be designed for self -initiated 
evaluation that would contribute to decision making involved 
in state level program planning. A second was that it be a 
continually operative mechanic so that program plans might 
be adjusted wherever required by changes in the field situation. 
The model integrated evaluation and program planning and is 
called the Evaluation-Program Planning Cycle. 

The goal statements for each objective were formulated 
to permit measurement of the extent to which: (l) t.^rget 
•copulations of concern are being served and (2) local 
programs of vocational education assure program comprehensive- 
ness, relevance to desired program outcomes, quality, and 
assessabillty. 



The data sets were designed to provide quantitative 
indicators of: (l) target population needs, (2) the 
effectiveness of training outcomes, (3) school and 
community ch5U?acterlstics, and (U) the relationships be- 
tween vocational education .program processes and training 
outcomes • 

State level program planners require evaluation results 
to plan and redirect agency activities and to selectively 
allocate the personnel and financial resources required 
to accomplish these activities. The data were organized 
in ways that would have tha greatest usefulness for 
these management operations. Data were organi7ed according 
to: 

1. Program sectors — public, private. 

2. Program levels — secondary^ post-secondary, adult. 

3. Program areas-— trade and industrial, agricultural, etc. 

4. Facility types — area vocational school, secondary 
school. 

5. Student characteristics and experiences --equal 
opportunity data, cooperative work exp-^riences , etc. 

6 • Schools . 

Three efficiency factors received attention during the 
process of formulating the data set: 

1. The time frame required for collecting the data. 

2. The costs to the state agency of carrying out 
data collection procedures. 

3. 'Personnel required to manage the data collections. 

SOURCE: Starr, Harold, et al. A System for Statt. Evdluati^m - 
of Vocational Education. Research Series No. 58. 
Washington, D.C.: U. S. Department of Health, 
Education and Welfare, Office of Education, 
Bureau of Research, Hay, 1970. 

PROGRAM PLANNING PPBS (Program Planning Budgeting 
BUDGETING SIS^^ System) is one way that objectives 
(PPBS) have been use** in administration 

models. 

PPBS has four niajor phases; (1) planning^ which is 
concerned with generating program objectives; (2) progvfmdng^ 
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which is concerned with the preparation of alternative sets 
of activities and services to acnieve program objectives, 
(3) budgeting, which is involved with the formulation of 
detailed plans, accounting, and reporting; and (4) evaluatingj 
which is concerned with progress outputs and effectiveness 
of programs. 

This form of budgeting requires that projected outcomes and 
evidence of past results be presented along with budget requests • 
It rejquires that cost-benefit analysis be done to compare the 
efficiency of one proposal with another. 

At the federal level it involves: 

1. Appraisals and comparisons of various government 
activities In terms of their contributions to 
national objectives. 

2. Determination of how given objectives can be attained 
with a minimum expenditure of resources. 

3. Projection of government activities over an adequate 
time horizon. 

Comparison of the relative contribution of private 
and public activities to national objectives. 

5. Revisions of objectives, programs, and budgets in 
the light of experience and changing circumstances. 

The model was introduced at the federal level and has been 
adopted at state and local levels. 

SOURCES: Hovick, David, ed. Program Budgeting. 

Cambridge, Massachusetts: Harvard University 
Press, 1965. 

Alioto, Robert, and J. A. Jungherr. "Using PPBS 
to Overcome Taxpayers^ Resistance." In 

Emerging Patterns of Administrative Aa^ountdbility, 
Lesley H. Browder, Jr., ed. Berkeley, California: 
McCutchan Publishing Corporation, 1971, pp. 252-59. 

SYSTEM VARIABLES Another kind of modeling relates 
APPROACHES objectives to the program and/or 

other factors • 

iWEA One of the first major approaches 

Model to broadening the lyier model came 

when Taha and Saw^n Introduced a 
model developed by the Association fox Supervision and 
Curriculiun Develcpment of the National Education Association. 
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Taba and Sawin listed the following weaknesses of evaluation 
as carried on in the schools: (1) the objectives that form 
the basis for evaluation are usually too narrow, (2) the 
range of instruments and devices that are being used is 
often too limited, (3) the focus of attention has been on the 
end product rather than" on process and has resulted in 
inadequate knowledge of the processes by which the end products 
are attained, (^) results have often been interpreted with- 
out adequate information about factors which affect learning 
and achievement, and (5) results of evaluation haven't been 
adequately translated into curriculum decisions. 

They proposed four major categories arranged on the four 
sides of the square. Objectives are on the top of the 
square with evidence of pupil behavior pertaining to the 
objectives on the bottom. Evidence of teaching- learning 
operations and evidence of nonschool factors affecting leam'- 
ing complete the other two sides of the square. 

SOURCE; Taba, Hilda, and Enoch Sawin. "A Proposed Model of 
Evaluation." Educational Leaderehip, XX 
COctober, 1962), 57-59ff. 

Cube Hammond's model presents relationships 

Models in a three-dimensional ciibe. One 

dimension is the behavioral dimension 
with objectives classified into three types: cognitive, 
affective, and psychomotor. The other two dimensions are the 
inatructional dimension (organization, content, method, 
facilities, and cost) and the inetitutional dimension 
(student, teacher, administrator, educational spf^cialist, 
family, and community). The sub-^items within the dimensions 
represent types of variables that may need to be considered 
in setting objectives and in examining the effectiveness 
of programs in meeting those objectives. 

Armstrong f id his colleagues from the EPIC Evaluation 
Centei at Tucson use a similar cube. They say that the 
institutional, instructional, and behavioral elements all 
should be considered when objectives are written and tlieir 
attainment evaluated. In addition, they add the variables 
of measurement, time needed, and proficiency level. 

SOURCES: Hammond, Robert L. Evaluation at the Local Level. 

Columbus, Ohio: Ohio State University Evaluation 
Center, 1967. 

Armstrong, Robert, et al. "A Scheme for Evaluation." 
In Educational Acaountdbitity Through Evaluations 
E. Wayne Roberson, ed. Englewood Cliffs, New <^erse> 
Educational Technology Publications, 1971. 



181 



Information Nelson suggests that there are three 

Domaina main domains from which information 

about how a developing curriculum 
attains its objectives is gathered. They are: 

Ingredients Processes ^ Product 



Learning materials Teaching acts Student 
Teachers Leeu^ning activities Teacher 

Students School environment School 

Schools Parent 
Community 

SOURCE: Nelson, Orville. "The Evaluation System for the 
American Industry Secondary School Courses." 
Paper presented at the AVERA program at the 
AVA Convention in Dallas, Texas, December, 1968. 

TYPES OF The categorization of types of 

EVALUATION eve/iuation as used by the Office 

of Economic Opportunity is proposed. 
Those four categories are: program impact evaluation, progrcun 
strategy evaluation, project evaluation, and project rating. 

Program Impact Evaluation. Assessment of the overall 
effectiveness of a program in meeting its objectives or the 
relative effectiveness of two or more programs in meeting 
common objectives. The usual objective of program impact 
evaluation is to help policy makers in reaching decisions 
on program funding levels or on possible redirection of 
the program. 

Program impact evaluation depends on the definition and 
measurement of appropriate output variables and on the 
use of appropriate comparison groups. Environmental and 
process data aren't usually essential. 

Program Strategy Evaluation- Assessment of the relative 
effectiveness of different techniques used in a national 
program. The usual objective of program strategy evaluation 
is to inform program managers of the relative effectiveness 
of the different strategies or methods used by projects in 
the national program. 

Program strategy evaluation depends on definition and 
measurement of appropriate environmental t input t pr^'^.esSy 
and output variables selected on the basis of suitable 
analytic models* 
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Project Evaluation. Assessment of the effectiveness of an 
individual project in achieving its stated objectives. This 
form of evaluation is required in many federal programs and 
is often carried out by the project itself. Project 
evaluation requires measurement of the important output 
variables c.5 well as the use of comparison groups. 

Project Rating. Assessment of the relative effectiveness 
of different local projects in achieving program objectives* 
The usual objective of project rating is to provide program 
managers with information on the relative success of local 
projects operating within a national program. In most cases,' 
it will make sense to rate projects against one another only 
if they^re operating in similar environment. 

Project rating depends on definition and measurement of 
environmental variables and rela-uively inexpensive output 
measures (for excunple, measure of short^temi impact). 

Each of the four categories can vary in the amount of 
detail considered and the level of decision making involved. ' 
The relative importance of the four also will vary from 
program to program and over time, as will th*^ feasibility 
of conducting such evaluation. 

Feasibility should be examined before evaluation is 
attempted. Whether evaluation is feasible depends on: type 
of evaluation wanted, availability of theory and measures 
suitable for the type of evaluation desired, time available 
for eva'^uation, availability of suitable comparison groups, 
feasibility of collecting the required data on program 
participants and members of appropriate comparison groups. 
In general, evaluation is more "^sasible when inputs are 
tightly controlled or when the effect of the program can be 
expected to be "large" compared to that of other forces 
that will be operating. 

Monitoring is always feasible from a methodological 
standpoint. Feasibility of the four types of program 
evaluation depends on whether suitable measures are avail- 
able. Project: rating is usuariy feasible. Its main problem 
will be devising suitable systems for classifying the environ- 
:nents within which p^oj^-ts are operating. Program impact 
evaluation will usually be methodology feasible. 

It*s usually impossible or too costly to learn enough 
from the evaluation of one local project to say with confidence 
whether the project — or other forces— actually caused the 
effects measur^ed. For example, success may be due to the 
charismatic teacher # 
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Related activities that arenH accepted by the author as 
being evaluation include: 

Monitoring. The assessment of managerial and operational 
efficiency of programs or projects, through periodic site 
visits and other management techniques. The usual objective 
of monitoring is to give program managers impressiuaiatic 
data about how their projects are going., to see if they're 
being run efficiently, if they're following program guide- 
lines, if they have competent staff in individual projects. 

Reporting Systems. Routine reporting from state or local 
level isn't evaluation, but may furnish useful data on services 
provided, populations served, and costs of providing services. 
In some cases, it may Le justifiable to spend evaluation funds 
to defray state and local costs of providing such data, as well 
as follow--up data on specified samples of program or project 
participants, when the data are required to meet clearly 
specified evaluation needs. 

Cost Analysis. Program managers often have good information 
on the cost (to the federal government) of providing services 
through a program. Comparative analysis of costs by project, 
by groups of projects, or by program is essential in good 
evaluation as well as a valuable management tool in itself. 

SOURCE: Hholey, Joseph^ et al. Federal Evaluation Polioy: 
Sooial Evaluation by Pedaral Agenoi^e. 
Washington, D.C.: The Urban Institute, 1970, 
pp. 1-^9. 
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5.4 OBJECTIVES: CATEGORIZATIONS AND CRITERIA 
ORIGIN General education. 



ESSENCE Part of the difficulty in evaluating the attainment of 

objectives is the difficulty in setting and using objectives. 
Recent efforts recognize the importance of better under- 
standing the phenomena that are objectives. 

USE Particularly useful when people are having trouble setting, 

using, and evaluating the attainment of objectives. 

STARTING Examine the objective and establish its relationship to 

POINT other objectives. 

DESCRIPTION Because objectives are the central focus of this approach 

to program evaluation, the nature of their being must be 
well understood. Various attempts have been made to describe 
and categorize kinds of objectives, that is, develop 
models of tj^pes of objectives. 

OBJECTIVES AS Objectives may b*: considered aq a 

SYSTEMS AND system. They he,ye three interrelated 

PARTS OF SYSTEMS parts: (1) the content or the "what" 

of the objective; (2) the type of 
behavioi? or action expected in relation to that content or the 
"what's he supposed to do with it" component; and (3) the "who" 
and "how many" components that indicate who's supposed to 
attain the objective. The three parts interact together 
and depict the "whole" of the learning that's expected to 
occur . 

Each of these three parts hDok into other systems. The 
"who" and "how many" hook into the socio-psychclcgical en- 
tities that are learners. The content hooks into systems 
for analyzing and categorizing content. Gagn^, the authors 
of the taxonomies of educational objectives, and Bri-aner have 
been particularly helpful in clarifying how various types 
of content (concepts, facts, chains of specifics, etc.) 
relate to each other and constitute a knowledge or attitude 
system. Furst, Mager, and the taxonomists have also been 
helpful in seeing how various kinds of behavior fit together 
into a whole range of behavior. 

Understanding how the parts of the system making up an 
objective fit into knowledge, attitude, and behavior systems 
makes it easier to determine the best way to examin<» whether 
the change specified in the objective has occurred* 
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SOURCES: Purst, Edward J. Conetruating Evaluation Inetvments. 
New York: Longmans, Green, 1958. 

Mager, Robert F. Goal Analyeie* Felmont, California: 
Fearon Publishers/Lear Siegler, Inc., Educational 
Division, 1972. 

Ma^er, Robert F. Preparing Inetruational Oboeoti'OeB. 
Felmont, California: Fearon Publishers/Lear 
Siegler, Inc., Educational Division, 1962. 

Krathwohl, D. R. , B. S. Bloom, and B. B. Mc^ia. 
Taxonomy of Educatioml Objectives: T^^ 
Claseifiaation of Edx national Goals. Handbook 2: 
Affective Domain. New York: McKay, 196if. 

Bloom, Benjamin S., ed. Taxonomy cf Educational 
Objectives: The Classifiaatior of Educational 
Goals. Handbook 1: Cognitive i3omain. New York: 
McKay, 1956. 

Gagne"^, R. M. The Conditions Of Learning. New York: 
Holt, Rinehart and Winston, :.956* 

Bruner, J. S. Tcmrd a Theori' of Inetruction. 
Cambridge, Massachusetts: harvard University 
Press, 1966. 

Although written as an individual uMit, an objective seldom 
stands completely alone. Individual objectives usually are 
part of some type of network. They usually make up a net- 
work in the sense that a set of objeccives at the same levels 
are stated for a particular activity. Although defining 
separate things, there often is a definite relationship 
among them. For example, cognitive, psychomotor, and affective 
objectives are closely interwoven. Adults usually don*t 
bother to exert the effort taken to inaster content or perfect 
skills unless they believe in what they are mastering. 
Sometimes objectives at the same level involve sequence. One 
has to be accomplished before its possible to accomplish the 
next. Often a major objective involves a whole package of 
more specific objectives each of which is essential to the 
attainment of the larger objective. 

The nature of the interrelation5.hips must be understood 
to adequately evaluate whether objectives have been attained. 
Two major ways of looking at relationships are longitudinally 
and hierarchically. 
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LEVEL AND Suchman discusses the relationship 

TIME SEQUENCE of immediate objectives, intermediate 

OF OBJECTIVES objectives, and ultimate objectives. 

He points out that chains ojF objectives 
exist linking objectives at different levels. The chain includes 
the idealized objectives or broad mission statements at one end and 
specific tasks at the other. 

Suchman also stresses that assumptions underlying the 
validity of each level of objectives and assumptions regard- 
ing the relationship between levels may need to be rechecked. 

Taylor and McGuire model a procedure by which broad 
objectives become specific student outcomes. They use four 
stages in their model: (l) broad objectives derived from 
societal and professional pressures, (2) interpretations 
into behavioral objectives, (3) translation into forms 
suitable for the classroom environment, and (*♦) student 
outcomes. They view curriculum evaluation as including 
data of two kinds: (l) objective descriptions of the goals, 
environments, personnel, methods and content; and immediate- 
and long-range outcomes; and (2) recorded perianal judgments 
of the quality and appropriateness of goals, inputs, and out- 
comes « 

SOURCES: Suchman, Edward A. Evaluative Research. New York: 
Russell Sage Foundation, 1967. 

McGuire, Thomas 0. "Decisions and Curriculum Objectives: 
A Methodology for Evaluation." Alberta Journal 
of Educational P^-aearohs XV (March, 1969), 17-30. 

Taylor, Peter A., and Thomas 0. McGuire. "A 
Theoretical Evaluation Model." Manitoba 
Journal of Educational Research, I (1966), 
12-17 . 



These are only a few examples of the kind of modeling that 
must be done to understand the nature of objectives in a 
program. Program objectives are usually broader than are 
objectives that guide instruction. The relationships be- 
tween the two must be clear. 



KINDS OF OBJECTIVES Instructional models limit themselves 
OPERATING WITHIN only to objectives dealing with learners 
A PROGRAM behavior. However, a program may have 

additional kinds of objectives. For 
example, programs usually include: reeutte obdcctivea, 
teaaher objeativeB, and adninietrative or program maintenance 
objeativee. 
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Results Objectives. May be of several types. Two of 
the most common are: (l) learner^entered objectives that 
indicate what change is to take place in the learners as a 
result of instruction, and (2) society^entered objectives 
that specify what will happen to society as a result of 
program . 

Teacher Objectives. These objectives state what the 
teacher is going to do. They* re usually the other side 
of the coin from the learner objectives • They include 
such things as: helping the student distinguish between 
two methods of work or motivating the least responsive 
members of the group to take an active part. 

Administrative or Program Maintenance Objectives. These 

objectives are institutional in nature, are concerned with 
program maintenance or instrumental objectives that focus 
on the mechanism that provides the program • 

This category includes objectives dealing with improved 
procedures, increased enrollments, improved contextual 
features and personnel. 

All three types of the objectives are legitimate and 
should be evaluated for their attainment. 

Learner-centered objectives may be further examined in 
terms of their origin. Sources may include: 

1. Objectives set by the learner himself. 

2. Objectives set by the representatives of the expected 
learners • 

3. Objectives recommended by those who know the expected 
■ learners very well. 

Objectives recommended by a larger Society or a 
particular pressure group who want to see another 
group make certain changes. 

5. Objectives set by a s\ibject -matter expert in terms of 
what the learner should know about the subject. 

6t Objectives set by the *'think group" of an agency to 
carry out the expected mission of the agency. 

7. Objectives set by teachers from thin air or drawing 
from any of the other sources • 
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Just because an objective is phrased in terms of the 
learner doesn't mean that it's an objective that's originating 
with any of the learners who actually take part» Supposedly 
the^'r act of participation meakis that they accept the 
objectives that have been announced. However, that's not 
always the case. 

There may be several different and conflicting sets of 
learner objectives functioning in any learning situation • 
Some of them will be stated. Others won't. For example, 
the agency head may think that one set of learner* s objectives 
is guiding the program. The teacher may have developed a 
set of learner's objectives based on her experience with the 
discipline, and the learners themselves may have quite 
heterogeneous sets of objectives neither agreeing with what 
the teacher has set or with each other. 

SOURCE: Steele, Sara M., and Robert E. Brack. "Evaluating 
the Attainment of Objectives: Process, Problems 
and Prospects." [Manuscript in process.] 

ADDITIONAL 

REFERENCES: Uhl, Norman P. Identifying Inatitutional Goals. 

Durham, North Carolina: National Laboratory 
for Higher Education, 1971. 

Peterson, Richard E. The Crisis of Purpose: 
Definition and Uses of Institutional Goals. 
Princeton, New Jersey: Educational Testing 
Service, 1971. 

CHARACTERISTICS Another way of getting a feel for 

OF OBJECTIVES the phenomena that's called objectives 

is by identifying the characteristics 
of objectives and trying to see how those characteristics 
affect their use and the evaluation of their attainment. 

Houle defines objectives as the actual intended result 
of learning and not just a mere formal expression of that 
intention. He suggests that objectives have the following 
attributes that affect programming. 

1. Objectives are essentially rational. 

2. An objective is practical. The ultimate test of an 
objective isn't validity but achievability. 

3. Objectives lie at the end of actions designed to 
lead to th«m. 

4. Objectives are usually pluralistic and require the 
use of judgment to provide a proper balance in their 
accompllsbment • 
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5. Objectives are hierarchial. 

6. Objectives are discriminative. 

7. Objectives change during the learning process. 

Warner refers to the characteristics of intangibility, 
change, number 9 continuousness, and remoteness as being 
barriers to using objectives effectively in managing programs. 

SOURCES: Houle, Cyril 0. The Design of Education. 

San Francisco: Jossey Bass, Inc., 1972. 

Warner, Keith. "Problems in Measuring the 
Goal Attainment of Voluntary Organizations." 
Journal of Adult EduaaHon, XIX (No. 1, 1967). 



GROUP 6 

Results-Devaluation of Outcomes and Effects 

OVERVIEW: 

How do you see beyond your objectives and look at what else is happening 
as a result of your programs? A growing number of approaches are dealing 
with outcomes and effects thai: include but go further than just looking 
at results related to the specific objectives of the program. These types of 
evaluation strategies are concerned with anticipated and unanticipated results > 
direct and indirect results, and benefits and harmful consequences. They 
vary considerably in their approach and coverage. Some of the material 
included in this section "model" the nature of results, some deal with 
procedures for examining results. 

EXAMPLES INCLUDED: 

# Goal free evaluaHon compares the results achieved with the original 
need as a way of establishing program effectiveness. 

# In the zonee of 2*eeulte approach, tliree sources of information about 
results are identified and data-^ga chering devices that can be used 
most effectively are discussed. 

• The multiple dimnaione of program effeoHveneee approach identifies 
various ways effectiveness can be defined. 

• The multiple change approach classifies kinds of results expected 

in community development programs and indicates sources of information. 

• Effectiveneee of methods provides an example of how results 

data were usee to examine the "payoff" of various kinds of programming 
methods . 

0 Impact eViXluo:tion looks at the long-^range achievements of a program. 

# Social indicators are used as a means of measuring community progress. 

# Institutional cvaZuation provides a framework for evaluating 
inulti-cours<3 curricula. 

• Public policy approach provides a framework for looking at the 
relationship of programming to piibllc policy. 

0 Research models review a traditional approach to results evaluation* 
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• Efficiency examinatione discuss the role of cost-benefit analysis 
" of results • 

• Acc'ountdbitity deals with the concept of stewardship and identifies 
related activities. 



GESERAL ADVANTAGES: 

0 Examination that includes a full spectrum of results gives us 
a better understanding of the actual effects of programs* 

# Such approaches deal with the corq>lexity and dynamism of human 
nature • 

O Examining results in a systematic way helps us become more realistic 
in our expectations of programs. 



GESERAL CAUTIOUS: 

# Because of the extensive range in results » when resources are limited, 
some means such as original problem or need, mission of the agency, 

or prediction of the most crucial positive and negative results, must 
be used to narrow the scope of results data that will be gathered. 

# Standards are needed for interpreting what amount or results it*s 
reasonable to expect from various amounts of program inputs with 
participants of selected characteristics (examples; task oriented 
compared with socially oriented; less than high school education 
compared with college graduates). 

jt. Clear patterns of organizing and managing results data must be 
set up well in advance of collection. Otherwise, the data 
I may become fragmented and not build a clear and coherent picture. 



192 



6.1 GOAL FREE EVALUATIOH 



ORIGIN 
ESSENCE 



Evaluation of educational innovations* 



Results of programs cu?e judged against the originating 
need. Cost of producing those results is considered in 
relation to costs of alternatives. 



USE 



Particularly helpful in situations where objectives are 
not clear or are unrealistic. Helpful too as a means of 
examining objectives to see if they limited the potential 
of th^i program) that iSf is it able to produce more than 
was originally expected from it? 



STARTING 
POINT 



Examine a profile of the needs that the program is addressed to. 



DESCRIPTION. One of the more often talked about and controversial 

models emerging in education is goat free evaluation^ 
GFE. Such evaluation starts without referring to the 
objectives of the program. If they are used, they* re used 
after the data are in to see which of the results that, 
emerged actually were intended. The base for determining 
what data to collect is a profile of identified needs. 
Actual effects are then compared with these needs rather 
than with the objectives. The model emerged through experience 
in evaluating products of research and development centers 
where it was discovered that sometimes the side effects 
were of greater value than the intended effects. 

The work of evaluation is viewed primarily as an 
activi-fy of condensation that includes two major stages — 
compveaBion and aredentiating. A mass of data and 
observations about a program are condensed through various 
means (compression) until judgments as to the value and 
worth of the program can be made (credentialing) . 

The steps in the process have been identified as follows; 

1. Characterizing — explore questions like: What is it? 
What*s the situation? What*s actually occurring? 

2. Clarification-^•-specify type of conclusions wanted: effects? 
Comparative effects? 

3, Causations-dealing with effects. 

4, Comprehensive check of consequences — ^pick up side effects. 
5# Criteria-Dualities that are meritorious. 
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6. Costs—was it worth money that went into it? Is it 
more or less expensive than other ways of getting 
the same results? 

7. Critical comparisons and competitors— -do cost analysis 
on them also* 

8. Credentialing — refer back to need assessment, attach 
merit, inferential step, putting stamp of approval 
on it. 

9. Conclusions — as identified in #2. 

Evaluation of effects may be concerned with any or all of 
the following: 

Immediate Perfomance'^^Changes In 

1« Knowledge. 

2. Canprehension or understanding. 

3. Motivation (attitudes, values, effect). 

4. Nonmental abilities (perceptual, psychomotor, 
motor, social skills)* 

5. Long-Grange changes* 

Secondary Effects On 

1. The teacher* 

2* The teacher's colleagues. 

3« Other students. 

4. Administrators* 

5 . Parents . 

6. The school* 

7. Th6 taxpayer* 

If, when the data are examined, results that coincide 
with the stated objectives are stronger and more conclusive 
than other results, this is particularly good evidence that 
the program was well designed to attain certain objectives* 
However, other results ar'enH lost in the process. 
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Evaluation is seen as an activity by which performance 
data are gathered and combined with a weighted set of 
goal scales to yield either comperative or numerical 
ratings, and as the justification of the instruments, 
goals, and ratings. Cost analysis is essential to explain 
the basic question of whether it was worth the resources 
that went into it. 

The goal of such evaluation is the answering of questions about 
value or worth. Usually, this goal is accomplished more 
effectively if comparisons are made between alternative 
programs rather than by comparing participants with those 
who haven* t participated in any program of this type. 

Scriven, Michael S. "The Methodology of Evaluation." In 
Perapeotivee of CuvHaulum Evaliiqtion, Ralph W. Tyler, 
Robert Gagne', and Michael Scriven, eds. AERA Monograph 
Series on Curriculum Evaluation, Mo. 1. Chicago: Rand 
McNally 6 Co., 1967. 

Scriven, Michael. "Prose and Cons About Goal-Free Evaluation." 
Eiyaluation Coffwent (December, 1972), III. 

Notes from AERA Concepts of Evaluation Workshop, Portland, 
Oregon, October 19-20 ♦ 1972 
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6.2 ZONES OF RESULTS 
ORIGIN Adult education. 

ESSENCE Results need to be examined within the learning setting, 

in real life during the course of the lea" ''^«> M in 
real life at an interval after the coinpl o Jxie 

learning activity. 

USE Helpful in identifying sources of information. 

STARTING List the kind of results expected at various times. 

POINT 

DESCRIPTION '^Formative evaluation'^ discovers whether the instructional 

methods are producing the desired result on course, providing 
information on what works with whom and what additional train- 
ing is required* When a prototype is being developed, it 
reveals what revisions in the coarse must be done to produce 
the desired result (for examp7e, lesson changes, sequencing 
changes, added lessons, substituted lessons, needed training 
techniques, etc.) and other modifications while the course 
is going on. ''Surmative evaluation^' provides, information on 
the total impact of the course.- 

This model for examining the results of a program identifies 
three main zones that are relevant in programs for adults. 
They are: 

Zone 1 --Study of Results As Evidenced Within the Class or 
Program Activities, simulation, role playing, and gaming 
a*.e used to trigger evidence of results within the actual 
program. 

Zone 2~App1i cation in Real Life, information is secured 
about how adults are using information in real life as the 
program of course progresses. 

Zone 3"Fonow-Up Study, interview and evaluation 
procedures are used at various periods after the course* 
The extent of success of transfer to real life while in the 
cotirse and after the course should be compared. 

This approach is particularly interested in the development 
of programs that will be distributed and used in many loca- 
tions and as result focus on results at various stages in 
the development process. The following questions are suggested 
as a guide to further development. 
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a. Which of the behavioral changes specified for the 
students actually occur as a result of the course? 

' b. What behavioral changes actually occur as a result 

j of the course, but aren't specified in the objectives'* 

i 

j c. In what respects does the course fail to provide the 

, opportunity necessary for the students to achieve the 

j specified behavioral changes? 

! 

d. What changes specified for the course are a function 
of time (no training)? 

' e. Does academic upgrading only result in the same be- 
havioral changes as these specified in the course 
combined with a sequential course? 

f . Assuming that students have expectations for a 

"traditional learning setting," will the provision 
of such a setting result in more efficient develop- 
ment of specified behavioral change? 

g* How do the techniques of skill training produce 
specified behavioral changes more efficiently than 
those techniques in the present course? 

h. In what ways does the precision required for the 
implementation of the techniques of skill training 
refine the definition of the behavioral objectives 
in the course? 

i. In what ways do the written lesson materials fail to 
give adequate guidance to the coach for effective 
lesson implementation (timings, sequence of lessons, 
clarity of direction, completeness of direction). 

j. What skills do the coaches lack to objectively 
achieve behavioral change in students? 

The model describes a variety of kinds of data and means 
of securing data in relation to one or more of the zones. 
Each description of a data collection approach has four parts: 
purpose, criteria, decisions,, and procedure, as illustrated 
in the following example that discusses direct obaervation 
as a method of securing data. 

Purpose n To identify behaviors that occur in response 
to specific lessons, teachers, and other student behavior. 

Cr1ter1ll. The behaviors must be observable: word, 
movement » facial expression t body attitude > associations 
with other persons in the learning group. The observer 

O 
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doesn't rate the quality of a behavior; he notes only its 
presence and in the report offers an interpretation of the 
behavior . 

Decisions. 

1. The teacher may use these records to modify his teacher 
behaviors to bring about desired changes in his group. 
He uses them immediately, 

2. The lesson developer uses the records to substitute 
new directions in the written lesson materials or to 
add new directions if he can establish the relevance 
between the noted behavior and the lesson* 

3. The observations provide the empirical basis for 
the description of the theoretical course model. 
These observations will determine the retention and 
modification of the model. 

Procedure. All training groups are observed through all 
lessons by trained observers. A behavior analysis that 
reflects behaviors expected in the lesson is used as a guide 
by the observer. He records a summary of behaviors exhibited 
by each group member at the end of each session. 

Lamrocky A. L., A. D. Smith , and P. W. Warren. ^'Evaluation: 
Its Scope and Systems for Evaluation Development." 
Paper prepared for meeting of research directors of NewStart 
Corporations 9 Ottawa, Saskatchewan » March, 1971. 
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6.3 MULTIPLE CHANGE APPROACH 



ORIGIN 
ESSENCE 



International development programs. 



Four kinds of results* six dimensions, and four major 
sources of information are outlined. 



USE 



Particularly useful in programs with goals that go 
beyond the particular participants. 



STARTING 
POINT 



Determine the type(s) of results that can be expected from 
the program. 



I 



DESCRIPTION This model was designed for examining international 

development programs. 

It suggests that several kinds of changes are needed 
for development to occur — payohological changee in 
motivation and attitudes; educational changee in know- 
ledge, concepts, and skills; technological changee^ new 
methods of production, organization, administration, and 
distribution; and Booiological changee^ relations among 
persons or groups. 

The model distinguishes between first approach or the 
immediate objective of the program, and more fundamental 
changes brought about as a result of the first approach. 
For example, a project that focuses on use of fertilizer 
might De evaluated only on whether the farmer applied more 
fertilizer this year* Progressively more fundamental, 
however, might be the following types of changes that might 
come from the program. Does he buy fertilizer at commercial 
prices in subsequent years? Is he more willing to try other 
new practices? Is he more willing to support or cooperate 
in the development of new agricultural techniques? 

Kinds of results sought by programs can be classified as 
follows : 

ImpeTBonal reawZta—legislation adopted, a remedy for a 
disease developed* 

Adminietrative reeulte — policies or procedures changed > 
institutions fitrengthened. 

Reeulta involving pereona; 

I. Changes in Information, attitudes, and skills* 



ERIC 




199 



2. Actions and habits of individuals or groups* 

3. Changes in social structure or relationships. 

Changes in eaonomCc and eocial conditiona veeuUing from 
actions taken by individuate or groicpe. 

Changea in capital reeouroee. 

The standards by which development projects should be 
judged successf»s or failures may be considered in terms 
of six major dimensions: 

1# The kind of results sought (including their 
permanence and spread)* 

2. The side effects produced. 

3. The area of activity and the area of influence 
where the results appear. 

'4. The time period within which results of specified 
size are obtained — the pace of change. 

5. The cost of bringing about results of specified 
size-^-^the efficiency of the project. 

6. The amounts of desired results brought about 
by factors other than the development project. 

One of the challenges is selecting which changes will 
be studied in light of the significance of those changes 
and the availability and reliability of data related to 
the changes. 

Although the intention and specific objectives of the 
project are a starting point, an attempt should al^'O be 
made to predict and examine potential side effects. 

Four major types of data are suggested: (1) already 
available data^ (2) additional record that organizations 
or individuals might agree to keep, (3) data obtained by 
eyetetmtic observation of behavior, and (U) data obtained 
by syetematic queationing. 

The obtaining of measurements of project results and 
costs isn't in itself evaluation. They have to be interpreted. 
Interpretation ia the conponent that diatinguiahee evaluation 
from maaurenient. 



SOURCE Hayes, Samuel P*, JrV Evaluating Development Projecta. 

Place de Fontenoy, Paris; United National Education, 
Scientific S Cultural Organization, 1959. 
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6.4 MULTIPLE DIMENSIONS OF PROGRAM EFFECTIVENESS 



ORIGIN 
ESSENCE 



University Extension. 



At least six dimensions of program effectiveness may 
need to be considered — general reeultey attairvnent of 
objeativee^ intent of the progrcan, impact and eignifiaanaes 
contribution to mCeeion, and use of reeouraee. 



USE 



Particularly useful in selecting results data that may 
be influential on outside audiences* 



STARTING 
POINT 



Review of the mission and intent of the program. 



DESCRIPTION All programs need to be examined in terms of their 

effectiveness. Effeotiveneae deals with their degree of 
eucoeee in producing reeutte. Effectiveness may have 
several components. 

One model proposes that there are at least six dimensions 
of effectiveness that should be applied to programs: 

1. General reeulte. 

2. Attainment of objeativee. 

3. Intent of the program. 
^. Impact of eignifiaance. 

5. Contribution to mieeion. 

6. Uee of reeouroee. 

The two dimensions that aren't covered by the other models 
in this section are contribution to mission and the intent of 
the program. 

Contribution to Mission. Most agencies and units within 
agencies either are commissioned by legislation or other 
enabling acts to a certain mission, or develop an overall 
mission within which they operate. The results of program 
should be analyzed in terms of the extent to which the 
mission is actually being accomplished* 



201 



Intent of Program. Host adult education programs can be 
categorized into certain types • Some have the traditional 
intent of diffusing knowledge and helping people to master 
content. Others are devoted to problem coping. Still 
others are concerned with developing generalizable skills 
in people such as abilities in analysis, creativity, 
or ability to communicate effectively. Still others are 
intent on changing conditions. Ultimately, a program *s 
results should be examined in terms of the basic intent 
of the program. If the intent of the program is problem 
solving or coping, the basic examination should be of what 
resulted in terms of the problem. Was it solved, eased, 
or controlled? If so, the program was effective in realizing 
the intent of the program. If a lot was learned but nothing 
done in terms of the problem, the program may have been 
successful on other dimensions of effectiveness, but failed 
in terms of its intent. Although things in addj/cion to the 
intent may have considerable value, part of the definition 

success must lie with the overall purpose for which it 
was undertaken. 



SOURCE Steele, Sara H, Six Dimemicma of Program Effeotiveneee. 

Madison, Wisconsin: University of Wisconsin^Extens^on, 
Division of Fro{p:>am and Staff Development, 1972, 
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6.5 EFFECTIVENESS OF METHODS 



ORIGIN 
ESSENCE 



Cooperative Extension. 

Data as to practices used and kinds of contacts with Extension 
were correlated to determine the effectiveness of various 
Extension methods. 



USE 



Useful when there's quantitative results data and clearly 
distinguishable methods. 



STARTING 
POINT 



Examine the natxore of practices and results to see if 
such a comparison can be made. 



DESCRIPTION 



Cooperative Extension took a nationwide approach to the 
examination of the comparative effectiveness of various 
teaching methods during the 1940s. Although the procedure 
hasn't been used recently except in one or two states, the 
basic pattern is presented because of its adaptability. 

First, data were gathered on Extension's influence on 
the use of farm and home practices. Data were gathered* 
from respondents from more than 15,000 farms and homes 
in 32 sample areas in 27 states'. Only the practices 
that respondents could associate with Extension teaching 
were included. Data were then analyzed to determine the 
number of practices that a particular method, demons traHona^ 
gonerat meetinge, office oalU, hulUtins, etc had 
influenced. 

Further analysis examined the relationships of methods 
to content of the practice. 



SOURCE 



Wilson, Meredith C, and Gladys Gallup. Extension Teaching 
Methods. Extension Service Circular 495. Washington, D.C.: 
Federal Extension Service-USDA, August, 1955. 
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6.6 IMPACT EVALUATION 



ORIGIN Manpower training programs* 

ESSENCE Impact evaluation looks at the ultimate effects of programs. 

USE Particularly useful in examining the extent to vhich an 
agency is accomplishing its mission. 

STARTING Examine the mission of the program and identify the major 

POINT results that program is expected to produce. 

DESCRIPTION Developed as a means of evaluating manpower projects, this 



model is projected by its authors as one of three approaches 
to evaluation. The other two are pTOoeat monitoring that^ 
examines the efficiency of a specific project and evaluation 
of the irmediate eucoeee or failure of a particular program* 
In comparison to them, intpaot evaluation examines the 
long-run goals of the program and views success and failure 
in these terms. 

The goal of such evaluation is to provide policy makers 
with the basic data necessary for them to make decisions 
wisely. Impact evaluation should provide five essential 
sets of information: 

1. All the data necessary to determine whether a particular 
type of program should be continued. 

2. Indication of which alternative programs provide the 
greatest gains for a given cost. 

3* Information on the component of each program and the 
mixes of components that are most effective for a 
given expenditure so that minimum operating efficiency 
can be achieved* 

Information for persons with different characteristics 
so decision makers can determine which individuals 
are best served by each program. 

I 5. Information about new methods of attack on this type 
of problem. 

The types of people who will benefit should be considered 
when criteria for success are being established. In most 
projects, two types are always present— the participants 
and society as a whole. In other instances, other types 
like employers, government, etc., are beneflolaries. Criteria 



of success should be developed from the standpoint of all 
that stand to gain or lose through the project. 

In each instance, the criteria for success need to be 
developed into specific goals with clear operational criteria. 

Another description of impact evaluation describes it as a 
term used to indicate the broad effects of a program. It 
focuses on the mark that a program leaves on the people, 
community, problem situation, and/or conditions within which 
the participants operate. It isn*t directly concerned with 
specific results of the program, but it*s concerned with 
how these changes affect other people or change conditions 
which affect people. 

Although most programs have some degree of impact, impact 
evaluation is concerned with examining whether the program 
has accomplished something of both magnitude and importance. 

This type of evaluation is best done on: 

1. Large-^cale projects. 

2. Projects of considerable force, concentrated intensive 
input . 

3. Longitudinal and J.ong-range programs. 
Combinations of efforts. 

a. Related program acti'^dties over several years. 

b. Efforts of a team of people. 



Borus, Michael E., and Villiam R. Tash. Measuring the 
Impact of Manpower Program: A Primer. Ann Arbor, 
Michigan: The University of Michigan — Wayne State 
University, Institute of Labor and Industrial Relations, 
1970. 

'^Impact Evaluation." Material prepared as a follow-up 
of a University of Wisconsin-JExtension Workshop on 
Impact Evaluation, Madison, Wisconsin, March » 1972. 



\ 



205 



6.7 PUBLIC POLICY 



ORIGIN 
ESSENCE 



General education. 



Program outcomes should be considered and classified as to 
whether they're just prog5raimnatic or whether they have 
public policy consequences. 



USE 



Particularly relevant to programs affecting disadvantaged 
adults • 



STARTING 
POINT 



Examine the effects of education on public policies. 



DESCRIPTION In addition to dealing with progranunatic questions, 

evaluation should contribute to better understanding 
and dealing with public policy questions. Programs include 
outcpmes relating to public policy when they J 

1. Directly or indirectly alter the power relationship 
between the citizen and the state . 

2. Affect immediately or in the long run the status a 
person has and the power he can exercise within the 
social system. 

3. Increase or decrease social tensions. 

4. Effect a change in the self -concept or sense of 
self-^Korth of the individual. 

The model identifies ways of categorizing outcomes 
in terms of the following table: 





Intended 


Unintended 


Outcomes 


\ealms of Outcomes 


Outcomes 


Anticipated 


Unanticipated 


Public policy 






Programmatic 







Intended public poliay outaoma are similar to the broad 
goals of education. Progranrnatio intended outcomes are the 
specific teaching objectives. Usually, the outcomes are intended* 
However, often programs also produce unintenticnat outcome. 
Some of them, although not sef as objectives in advance, may be 
anticipated. Others are completely unexpected. 

Public policy questions relate to moral Vatuee. 
Evaluation that deals with public policy questions must 
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be able to provide ihformation related to the raoral value 
questions that are involved. However, models for dealing 
with morc^l issues are lacking. 

SOURCE Berlak, Harold. ''Values, Goals, hiblic Policy and Education- 

al Evaluation." Review of Eduoational Reeearohs XL 
(April, 1970)^ 261-78, 
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6.8 INSTITUTIONAL EVALUATION 



ORIGIN 
ESSENCE 



Higher education* 



Performance on standards of student achievement and attitudes 
toward the program are key elements in examining the curriculum 
offered by a department or unit. 



USE 



Particularly useful when a sequence or package of courses 
and activities is involved. 



STARTING 
POINT 



Establish the general standards that graduates should achieve. 



DESCRIPTION Developed as a framework for evaluating programs in higher 

education, institutional evaluation is concerned with questions 
raised by administrators, suppliers of funds, accrediting 
committees, and internal and external critics. Questions 
like: Do the objectives underlying the program adequately 
••eflect the institution's goals? Does the benefit achieved 
by the program justify the expenditure of resources? Are 
the institution's objectives satisfactorily met; if not, 
should the institution demand modification of the program or 
should it modify the goals? In short, is the program 
acceptable to the institution? 

On the other hand, project evaluation takes place 
within the process of curriculiam design and deals with questions 
posed by the originating professor or design team* 

Both project and institutional evaluation are concerned 
* with the performance of students. .However, project evaluation 

is concerned with performance in terms of the objectives of 
the course while institutional evaluation is concerned with 
performance in terms of general standards for the particular 
type of student. 

Institutional evaluation generally focuses on the program 
as a whole rather than on its constituents. It considers the 
achievement of any particular program in relation to a network 
of other programs and goals, a function that can't be done 
at the project level. 

Institutional evaluation uses two major kinds of data: 
information on aofiievemnt of stofularde of student achievemnt 
and information about attitudes toward the program. 



A constructive system of university standards is based on 
the following premises: 
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!• Standards are subject to continual revisj.on. 
They inust be sensitive to changes within fields 
of knowledge and in educational philosophy* 

2. Standards result from a process of analysis that 
includes input from a wide range of opinions and 
perspectives within ah institution. 

3. Standards are developed in cognizance of, though 
not necessarily in agreement with, the objectives 
reflected in the design of relevant programs. 

Standards may be best set by committees that include 
members of the department originating the program, members 
of the departments where proficiency is needed f^r their 
own course work, and other knowledgeable faculty mbers 
serving at large. The committee would: 

1. Study objectives and procedures of a course being 
developed. 

2. Examine criteria that define what the particular 
type of student ought to be able to do in the subject 
area (opinions of relevant faculty members, opinions 
of professionals outside the organization, specifica- 
tions developed by profisssional associations, re- 
quirements of upper level courses, requirements of 
employment) • 

3. Develop a statement of institutional standards in two 
forms: 

a. A conceptual statement identifying concepts and 
skills that are important parts of achievement* 

b» An operational statement including standards of 

achievement and the way in which they will be examined* 

The standards examination wouldn't be a course examination 
used to evaluate the performance of individual students, but 
would be used periodically to check the performance of the 
courses of the department. 

Attitudes Toward Program. Reactions of students and 
faculty serve to indicate the degree of a program's acceptance. 
Opinion polling methodology may be used to secure opinions 
from students who have participated in the program, students 
scheduled to participate, and faculty of relevant departments. 
Opinions about the objectives of the program, effectiveness in 
meeting the goals t and opinions about continuation of the pro- 
gram would be gotten. 
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Integration of Information. Curriculum reports should 
be done with the same care as research reports. 

An evaluation study doesn't itself constitute the evalua- 
tion. Data must be interpreted and acted on by appropriate 
representatives of the organization. Evaluation isn't a 
single study or series of studies; it's an ongoing activity 
designed to provide answers to questions that arise from a 
number of sources and perspectives. The number of types 
of studies conducted at a given time depend on the institu- 
tion's needs and resources devoted to evaluation. Selection 
of appropriate studies is an important part of the work of an 
evaluation program. 

The information will be useful only if it does in fact 
contribute to the institution's appraisal of its own effective- 
ness. Judgment by responsible organizational members is 
required. Curriculum evaluation requires the ^ ^rticipation 
of persons from all parts of the institution. An institu- 
tion-wide Curriculum Review Committee is recommended. 



Forehand, Garlie A. "An Evaluation System for Curriculum 
Innovation." Tecuyher^a College Records LXXII (May, 1971), 
576-91. 
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6.9 SOCIAL INDICATORS 



ORIGIN 
ESSENCE 



Social planning. 



Programs dealing with visible entities that are recorded 
in piiblic data can use those indexes to plan and evaluate 
programs • 



USE 



Use is restricted to those types of programs where it*s 
possible to usa social-vindicator type data. 



STARTING 
POINT 



Identify the essential attributes that are within the 
scope of the program. 



DESCRIPTION Those agencies carrying out broad-based, many-faceted 

programs in a community might want to consider measuring 
results in terms of social indexes. The Community Health 
model, for example, uses measures on indexes in political, 
environmental^ economic, social, and educational fields to 
analyze community situations as a basis for planning and 
uses follow-up measures as a means of examining results. 
Each of the major areas is examined by means of a group of 
subscales. For example, one of the subunits in the political 
area is participation in the electoral process. It*s 
examined in terms of such data as the percentage of the 
adults registered to vote, the percentage that really do 
vote, participation by ethnic groups. 

A process for using social indicators includes : 

Step 2# Identify the essential variables that describe 
the whole system of reality essential to under- 
standing the meaning of a healthy community. 

Step 2. Identify the essential factors or subvariables 
that constitute a reliable description of each 
variable . 

Step 3. Determine the most appropriate forms for measuring 
them-^-that is, determine which social indicators 
will be used. 

Step 4. Develop norms or targets as standaK^s for defining 
a healthy comm"-nity. 

Step 6. Determine how citizens in the community define 
well being in the context of the variables and 
subvariables. 
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Step 6. Make a comprehensive examination of the 
community's own reality. 

Step 7. Analyze existing pxiblic and private programs. 

Step 8. Determine what should be done. 

Step 9. Determine the costs of alternative programs. 

Step 10. Assign program responsibility.. 

Step 11. Design mechanisms for evaluating the program so that 
there can be regular assessment of the relation- 
ship between the normative standards that the 
community has set for itself and the existing 
reality . 



SOURCE Paulson, Belden. A Model for Conmmity Andlyaia: Steps 

in Planning the Total Health of a Conmmity. Milwaukee, 
Wisconsin: University of Wis cons inHExt ens ion, 
Center for Leadership Development, April, 
1972. 



ADDITIONAL Munson, Byron E. Changing Conmmity Dimeneione. Columbus, 
REFERENCES Ohio: Ohio State University, 1968. 

Etzioni, Amitai, and Edward W. Lehman. **Some Dangers in 
* Valid* Social Measurement." The Annals of the Amevioan 
Academy^ II (September, 1967), 1-15. 



Toward a Social Report. Washington, D.C.: U.S. Department 
of Health, Education and Welfare, January » 1969. 



6.10 RESEARCH MODELS 



ORIGIN 
ESSENCE 

USE 



Social science. 



The presence or absence of results can be detected 
by using standard research design. 



Of limited value in situations where program input is too 
low to produce major measurable results or external variables 
are complex and not subject to research controls. 



STARTING 
POINT 

DESCRIPTION 



Establish the dimensions of the study. 

Some evaluation models duplicate research models with few 
if any modifications. Three research approaches to evaluation 
are given as examples: 

Longest suggests the following process: 

1. Define the general conditions and problems that 
require the evaluation to be made. 

2. Specify theories » assumptions, and values that are 
accepted as true and will guide the formulation of 
the evaluation research. 

3. Specify exactly what it is that will be evaluated. 

4. Formulat': I.ypotheses. 

5. Identify variables. 

8. Specify units of analysis. 
7. Collect data. 

d. Analyze data. 

9. Plan for reporting. 
10. Plan for executing. 

Weiss identifies the following core elements: 

1. Formulating the program goals that evaluation will 
use as criteria. 

2. Choosing iroong miltlple goals. 
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3. Investigating unanticipated consequences. 

^. Measuring outcomes. 

5. Specifying what the program is. 

6. Measuring program inputs and intervening processes* 

7. Collecting the necessary data. 

She deals with selecting crucial factors from among 
dependent, independent, and intervening variables. The 
depe:ident variables are the indicators of the program out- 
comes. The inputs are the independent variables. The 
intervening variables are the factors that mediate between 
the two. Adequate indicators usually involve multiple 
measurement . 

OutCOn>^ variables may include such things as: 

Effects on persons served. 

Effects on agencies. 

« 

Effects on larger systems* 
Effects on the public* 

Input variables include: 

Vvogvam varidbtee: purpose, principles, methods, 
staffing, persons served, length of service, location, 
size of program, auspices, management* 

Partioipant varidblea: age, sex, socioeconomic status, 
race, length of residence in community, attitudes toward 
the program, motivations for participation, aspirations, 
expectations from the program, attitudes of other 
family mendbers, degree f support from others for the 
intent of the program. 

Intervening variables Include: 
Progrcon^operation varidblee. 

Bridging vaHdblee: attainment of intermediate mile- 
stones # 

Developing a picture of the intended processes of a program 
may be helpful in examining which variables to use or control* 
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Although it uses program goals as a central focus, it 
also recognizes the importance of examining unanticipated 
consequences • 

The foregoing examples apply a deductive research 
approach. A third, less well-formalized approach draws 
on inductive approaches to research. Tentatively labeled 
"search" evaluation by Cain and Hollister, it sets out 
to determine what happens during and as a result of a 
program. However, the authors are more prone to recommend 
use of intentional experiments* 

SOURCES Longest, James W. "Designing Evaluative Research." Paper 

presented at the annual meeting of the Rural Sociological 
Society, San Francisco, California, 1969 • 

Weiss, Carol H. Evaluation Reeeax^^h. Englewood Cliffs, 
New Jersey: Prentice-Hall, Inc., 1972» 

Cain, Glen G#, and Robinson 6. HoUister* HUhe Methodology 
of EvaltuxHng Social Action Programe. Madison, Wisconsin: 
University of Wisconsin, Institute for Research on Poverty, 
n*d., pp* 9 and 42-^9* 
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6.11 EFFICIENCY EXAMINATIONS 
ORIGIN Industry and the military. 

ESSENCE Evaluation of the amount of benefits in comparison with 

the costs of the benefits and with alternative investments 
of the same resources. 

USE Particularly useful when time 9 money or energy resources of 

the participants or the programming agency are limited. 

STARTING Develop units that describe the costs and the benefits of 

POINT a program. 

DESCRIPTION Efficiency includes results but goes beyond them to 

examine the cost of those results. Examination of efficiency 
usually involves some type of cost-benefit analysis. 

Costs may be of two kinds: direct ooete and opportunity 
coete. Opportunity costs consist of other benefits forgone. 
Benefits also are of at least two tjrpes. The most commonly men- 
tioned types are private^ those accruing to the indivi4uals 
involved, and Boaiats those that acrue to society and not 
to one individual. 

There are usually five elements present in a cost-- 
benefit analysis: (1) the objective (or objectives), 
(2) alternatives for reaching those objectives, (3) costs, (^) a 
model (or models), and (5) a criterion. The criterion is 
the rule or standard by which the alteraatives are ranked 
in order of desirability and the most promising ones chosen. 

The process involved includes the following steps: 

1. Identifying the pertinent measures of effectiveness, 
that is, benefits. 

2. Describing the alternatives. 

3. Expressing both mission performance and cost as func- 
tions of the characteristics of each alternative. 

Estimating appropriate values for the equation parameters. 

5. Computing, analyzing, and presenting results. 

In some instances, cost^enefit is concerned only with the 
proportion of costs to benefits for the given program approach. 
Hora often f however, the approach is more useful if it deals 
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with alternative usage. It explores such questions as: 
Is there any better way to attain the same results? Can 
the same end result be achieved at less cost? Which of 
two approaches will produce most at least cost? Would 
there have been a greater value to the participants, to 
the agency, or to society if the same resources had been 
invested in a different program? 

Adult education programs need to consider the adult's 
resources and costs to peirticipants in terms of time and 
energy, and the resources of society that are being expended 
if the particular program is tax supported, as well as the 
usual concept of agency resource inputs. 

Some of the kinds of program outputs that may need to 
be considered are: 

1. Economic improvement— gain in econcwnic status of 
the participemts , conrounity, groupt or region. 

2. Action — outputs of direct action. 

3. Specific knowledge^ attitude, or skitl. 
^. Developmnt — personal or group. 

5. Enhanced eel f '-concept. 

6. Satiefaction and enjoyment. 

7. Preetige. 

Often outputs need to be transferred into some quantitative 
unit other than dollars. Tliis is sometimes done by establishing 
d. symbolic value unit and measuring the quantities of such 
units generated by a particular program. 

SOURCE Steele, Sara H. Coet-Benefit Analyeie and the Adult Eduaabor. 

Syracuse, New York: , ERIC Clearinghoxise on Adult Education 
and Adult Education Association of the U.S.A., 1971. 



217 



6.12 ACCOUNTABILITY 



ORIGIN 
ESSENCE 



General education. 



Programmers funded through tax monies are stewards of 
those monies and responsible for producing adequate results < 



USE 



Accountability receives greater stress the farther removed 
the program is from its source of funds and/or the larger 
the groups involved. 



STARTING 
POINT 



Analyze who your program is accountable to« 



DESCRIPTION 



Accountability is a process occurring between those 
entrusted with the accomplishments of specific tasks 
ietewarda) and those having the power of review irei^ietoere) • 
The heart of the process is for the party "standing to account"— 
the steward-^-to explain as rationally as possible the results 
of efforts to achieve the specific task objectives^of his 
stewardship. 

Reviewers will be concerned with the matching of performance 
and attainment levels against their expectations as expressed 
in task specifications and making a determination of their 
level of confidence in the steward and his efforts. 

The steward standing-to-^ccount will be concerned with 
his ability to accomplish the specified tasks and his ability 
to explain attainment levels in a manner that maintains or 
builds the reviewer ^s confidence in his stewardship. Various 
approaches to accountability are being advocated. One involves 
specific performance contracting on the part of teachers. 
Another increases the autonomy of local schools and builds 
increased local participation. However ^ instead of^ or in 
addition to, such specific activities, improved accountability 
is seen as having 12 critical factors. 

Community Involvement. The use of members of concerned 
community gr oups in appropriate phases of progreun activity 
to facilitate: program access to community resources; 
community understanding of the program's objectives, 
procedures, and accomplishments; and the discharge of 
program responsibilities to relevant community, clients service, 
and support groups. 

Technical ASSI stance • The means for providing adequate 
resources in program planning, implementation, operation, and 
evaluation by drawing on connunity, business^ industrial. 
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labor, educational, scientific, artistic, social/welfare, 

and governmental agencies for expertise and services necessary 

to effective operations* 

Needs Assessment. The identification of target-group and 
situational factors essential to the planning of a relevant 
program of action* 

Change Strategies. The development of effective strategies 
for systematic change in the educational enterprise and the 
incorporation of the strategies of program operations* 

Management Systems* The adaptation of the systems approach, 
through such techniques as Management by Objectives, PPBS, 
PERT, CPM to educational program management at the local, 
state, and federal levels. 

Performance Objectives. The specification of program 
objectives in a comprehensive, precise manner that indicates 
measures and means for assessing the degree of ati-ainment 
of predetermined standards • . 

Performance Budgeting. The allojation of fiscal resources 
in accordance with program objectives to be realized, rather 
than by objectives or functions to be supported • 

Performance Contracting. The arrangement for technical 
assistance in program operations through internal or external 
contracts where compensation is based on the accomplishment 
of specified performance objectives • 

Staff Development* The determination of the natiire and 
extent of staff development needed for the successful 
implementation of the accountability concept at the local, 
state, and federal level, and the design and conduct of 
indicated development activities. 

Comprehensive Evaluation. The establishment of systems 

of performance control based on the continual assessment 

of the program's operational and management processes and 
resultant products. 

Cost Effectiveness. The analysis of unit results obtained 
in relation to unit resources consumed under alternative 
approaches to program operation, as a determinant in continued 
program planning. 

Program Auditing. A pei^formance control system based on 
external reviews conducted by qualified outside technical 
assistance, designed to verify the results of the evaluation 
of an educational progran and to assess the appropriateness 
of program operation and manageiient. 
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The search for new ways of implementing the accountability 
concept in education can be expected to continue. The current 
range of ideaiSf practices, and definitions of educational 
accountability is broad and differing in rigorousness. But 
the pressure of the times and the advances of technology 
should be enough to ensure the continuation of the search. 



SOURCE 



Browder, Leslie H., Jr. Emerging Patterna of Adninietrative 
Accountability. Berkeley , California: McCutchan 
Publishing Corporation, 1971, pp. 1-21. 



ADDITIOKAL 
REFERENCE 



It 



'Accountability in Education." Eduoabionat Technology 
XI (January 1971). 
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CONDENSED CONTEMPORARY PROGRAM EVALUATION APPROACHES 



Appendix A 

CONDENSED CONTEMPORARY PROGRAM EVALUATION APPROACHES 



Below are condensations of contemporary progreun evaluation approaches* 
The upper-left comer includes the title and author of the approach • The 
upper-right comer lists the group number and page number where the approach 
is located in the text* The approaches are alphabetized by title* 



Accountability is a process occurring between those entrusted with the 
accomplishments of specific tasks » stewards » and those having the power of 
review, reviewers. The heart of the process is for the party "standing 
to account" — the steward-^-to explain as rationally as possible the results 
of efforts to achieve the specific task objectives of his stewardship* 



Evaluation systematically examines qualitative differences in program 
expectations of administrators and supervisors, teachers, students, and 
among these three groups by analyzing incongruities between intent and 
reported practices* Evaluation is focused on: staffing, recruitment, 
oolldboratim, in-service eduoation, goals, inatruotion, and the 
relationship among these oomponente. Each is seen as having inputs, 
processes, and outcomes, with the outcotoe of instruction being the ultimate 
focus* 



Appraisal Model 

Harris P# 135 

Appraisal is an act of judgment in which the judging implies both a 
criterion — a standard of some sort — and a pertinent description of what^s 
being done* The criterion and the observation must deal with the same 
thing. Aspects of a program that are subject to appraisal include: plane 
and purposes, resources, proceeees, and effects. 



Accountability 



6.12 
p* 217 



Actual Component Approach 

Knox, Mezirow, and Darkemald^ Jr. 



2*2 

p. m 
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Countenance of Evaluation 
stake 



3.1 

p. 120 



Evaluation data are either deeoriptive — intente and obeervatione — or 
judgmental. Both descriptive and judgmental data ara gathered about 
anteaedentej tvanaactions^ and outoomeB* Examining aontingenaiee and 
congmenciee among the sets of data Iz an important part of evaluation. 
For example, is what was observed congrsaent with what was intended? 



Criteria and Objectives 5,3 
Matteeon p. 17U 

Evaluation includes: puzpose of evaluation, criteria selection 9 
preparation for evidence collection, evidence collection, analysis and 
interpretation, reporting and use. Criteria areas include: effeativeneees 
efficiency J euitdbilitys and infpovtanae. 



Criteria of Success Approach 5.3 
Stichman p* 171 

Attention is given to a sequence of objectivee from immediate to 
intermediate to long range. An understanding of the aaeumptions that 
underlie the objectives and a checking of the validity of those 
assumptions is essential. 

Five general categories of criteria according to which the success 
or failure of a program in attaining its immediate^, intermediate-, or 
long-range objectives may be evaluated are: efforts performances 
adequacy of performomce, efficiency s and proceee. 



Cube Models 5.3 
Hantnond; Ametrong et al. P* 180 

Cube models show the relationship of program to the attainment of 
objectives. The three sides of the cube are: behavioral dimeneions 
inatructional dimeneions and inetitutionat dunenaion. Sub-^items 
within the dimensions represent types of variables that may need to 
be considered in setting objectives and in examining the effectiveness 
of programs in meeting objectives* 



Data Management 

Phi Delta Kappa 



U.2 

p. 138 



Most evaluation involves some handling of data. This approach to 
data management identifies four major elements: (l) delineating the 
information system needed, (2) obtaining the information, (3) providing 
the information, and (U) evaluating the evaluation. Soundness of the 
data rely in part on the soundness of the assumptions that are made in 
the data-handling processes. Six criteria should be kept in mind if 
evaluative data are to be informative to the receiver: i^levances 
importano0s eoope^ aredib-f-lity^ timalineee^ and pBrvaaiv^neeB . 



Decision Centered Evaluation (CIPP) i.i 

Stufflebeam; Phi Delta Kappa p. 78 

Evaluation provides information for judging decision alternatives. It 
can be useful at all stages of decision making- -aiMiwieeej deeign^ dhoioe^ 
and aotion. Fcur types of evaluation^-conteartj inputs proaeees and pro- 
duct — correspond to four major kinds of programming decisions — planning^, 
etruQturings implementing ^ and veayoling. 



Differential Evaluation 1.2 

Tripodi^ Felling & Epetein p. 81 

Evaluation is a management technique through which systematic feedback 
is used in improving programs. Programming ic examined in terms of the 
efforts invested;, effeotiveneee of those efforte^ and efficiency with which 
the effectiveness is achieved. In addition to the three types of examinations, 
three program stages are indicated: program initiation^ program contacts 
and program implementation. The term differential refers to the fact that 
the effort, effectiveness and efficiency questions asked are different in 
each of the three stages. Techniques used in doing evaluation are grouped 
into three categories : monitoring techniquee^ eocial research technique j 
and coat^analyeie technique • 



Discrepancy Evaluation 1.3 

Proms p. 84 

Evaluation identifies discrepanaies between actual programs and 
standards for programs so that progr 3 can be improved. Program personnel 
set standards for activities and rest .ts at each programming stage — design^ 
installation^ process^ and product. Actual performance is compared with the 
standard and discrepancies or areas for improvement identified* Discrepancy 
information is used either to change the performance or to change the 
standard* 



Effectiveness of Methods 
Witeon i Gallup 



6.5 

p. 2C'> 



Data as to practices used and kinds of contacts with Extension are 
correlated to determine the efft^ctiveness of various Extension methods. 



Efficiency includes results, but goes beyond them to examine the cost 
of those results. Examination of efficiency usually involves some type 
of ooet-'benefit analyeie. 

Costs may be of two kinds: diveot oObJB and opportunity ooete. 
Opportunity costs consist of other benefits foregone. Benefits also are 
of at least two types. The most commonly mentioned types are privates 
those accruing to the individuals involved, and social, those that acrue 
to society and not to one individual. 



Evaluating Decision Making 2.3 



This approach suggests that a matrix consisting of: (l) the problem 
requiring the decision, (2) a list of alterruxbive dedaiones and (3) a' 
generic liet of criteria used to consider each alternative decision be used 
as a means of analyzing the appropriateness of the actual decision made. 
Criteria are classified as: technological, Bociopeychclogicals and 
economic. 



Evaluation As Facilitation of Learning 5.2 
Bloom, Hactu^gss & Madaua p. 1S8 

Evaluation is a :.i3thod of acquiring and processing evidence needed to 
improve the student^s learning and the teaching. It*s a tool in education 
practice for determining whether alternative procedures are equally effective 
in achieving a set of educational ends. A two-dimeAsional figure shows 
relationships among analyeie of the leameros inetructional decieiones and 
evaluation and etudentes inetructional proceeeeos and objectivee. Three 
stages of evaluation are indicated: initial, formative , and eummative. 



Execution-Impact Approach 2.1 

Freeman & Sherti>ood ?• 102 

Included in a discussion of researches role in social policy formation, 
this approach suggests that the policy maker needs to determine whether: (l) the 
program was carried out according to prescriptions set forth in the planning 
and development stages— oaa^aament of execution, (2) the program worked — 00- 
eeeement of efficacy of impact , and (3) the expenditure of resources was the 
best considering alternatives^^-oaaMamnt of efficiency of impact. 



Efficiency Examinations 



6.11 

p. 215 



Heeaeling 



p. 115 
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Four-Question Approach 

Gottman <t Claeen 



5.1 

p. 156 



A recent text built around the Tylerian approach uses four questions 
as a means of organizing evaluation. They are: Vhy? What? BcW? and 
Bow idll you know? 



Reeulte of programs are judged against the originating need rather than 
the stated objectives. Cost of producing those results is considered in 
relation to ooete of altemativee. The work of evaluation is viewed as 
primarily an act of oondeneation that includes two major stages-^-oompreeeion 
and aredentialling, A mass of data and observations are compressed until 
judgments as uo the value or worth of the program can be made — credentials 
assigned* 



Goal -Referenced Instruction 5.2 
Popham & Baker P« i^^f 

Goal-referenced instruction has four parts: epeaifioation of objeotiveBs 
preaaeeeemnt^ instruction^ and evaluation. Two kinds of techniques can be 
used to judge the degree to which the learner has attained the instructional 
objective: (l) obeervation of behavior and (2) examination of produote 
that the learner will produce. Evaluation is an act of assessing merit by 
judgmentally comparing the observed result of some educational enterprise 
with a desired standard or criterion of acceptability. 



Impact Evaluation 
Borue & Taeh 

This model is projected as one of three approaches to evaluation. The 
other two are project monitoring that examines the efficiency of a specific 
project and evaluation of the immediate euaoeee or failure of a particular 
program. In comparison to them, impact evaluation examines the long-run 
goals of the program and views success and failure in these terms. 



Improvement Evaluation ^-5 

Kreitlou) P- 

The judgment of outsiders can play an important catalytic role in helping 
local programs identify areas where improvements need to be made, stimulate 
action in areas where improvements must be made, and reinforce confidence in 
those areas where the program excels. Those coninissioning outside appraisers 
need to clearly define the epeaifio purpoeee of the evaluation and establish 
prioritiee as to what will be examined. 



Goal Free Evaluation 

Soriven 



6.1 

p. 192 
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Information Domains 
Nelson 



5.3 

p. 181 



This model suggests three main domains— ingfrediente, proaeeeee, and 
product — from which information should be gathered in terms of how a 
curriculum attains its objectives. 



Inetitutional evalur/ion is concerned with questions raised by administra- 
tors, suppliers of funds, accrediting committees, and internal and external 
critics. Questions like: Do the objectives underlying the program adequately 
reflect the institution's goals? Does the benefit achieved by the program 
justify the expenditure of resources? Are the institution's objectives 
satisfactorily met? If not, should the institution demand modification of the 
program or should it modify the goals? In short, is the program acceptable 
to the institution? 



IPI Formative Evaluation 1^5 
Lindvall & Cox p* 89 

Formative evaluation is the aontinuoua evalvation of all elemente of a 
developing educational program as an aid to the development process. It 
deals with the basic question: How can every element and operation in the 
program be examined so it contributes to its improvement? Subquestions 
deal with the goals to be achieved, the plan for achieving the goa 3, the 
operating program, and the achievements of the program. 



Emphasis is on involving local people (staff and citizens) in making 
evaluation of the extent the program is reaching Its objectives. Essential 
elements include: commitment by local administration "j strategically placed 
local leadership team, active staff steering committee, functioning citizens' 
committee, allocation of time to the leadership team, training in research 
and evaluation procedures t communications and visibility, qualified consultant 
service. 



Institutional Evaluation 

Forehand 



6.8 

p. 207 



Locally Directed Evaluation 

Byram & Robertson 



5.3 

p. 176 
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Macro System Model 2.3 
Alkin P- 

Evaluation is the process of first identifying and then quantifying the 
relationahips between student inputs and eduoational outputs , given a constant 
financial input and controlling for effects of external systems. Components 
include: atutfent inputs; educational (Wtputa— changes in students and the 
impact of the program on home, community, other programs; financial inputs s 
mediating factors or the way the financial inputs invested in personnel, 
organization, instructional design, etc., are used in the program in combina- 
tion with student inputs; and external aj/atcme—social, political, legal, 
economic outside of the school that encompass the program and have impact on it. 



Management Information Systems (SEMIS) 

The Extension Management Information System is an example of computerized 
systems for pooling program data. The Extension system consists of records of 
planning^ activity reporting, and evaluation. The three sets of data are 
used as a basis for administrative and professional decisions. 



2.3 

p. m 



Means -Ends Hierarchy 

Bennett 

The model shows the relationship of se^en levels of evaluation data: inputs, 
activities, people involved, reaction, ^'KAS^' changes, change in practices, and 
ultimate results. The means-ends hierarchy provides a framework for identifying 
specific evaluation tasks and seeing them in the perspective of other tasks. 



3.3 

p. 129 



Monitoring Evaluation 
Bruce 

Monitoring evaluation deals with the critical management tasks in keeping 
programs on track. It^s continuous rather than episodic^ Major elements of 
the program included in the model are: time, program action, and expected 
traditional outcomes. Evaluation cycles consist of four stages; design, 
observations, judgment, and adjustment. 



4.4 
p. 146 



Motivational Model 2.4 

Y 

Motivation helps to explain huiran behavior. Analysis of motivation may 
be important in program evaluation. Education is concerned both with motiva- 
tion within (professional) and motivation without (the client)* In each 
instance, there •is input, motivational processes, and output. Motivational^ 
processes . are grouped . Into three csXegovlesx ' orgariiMticn, group integration, 
and personal. Evaluative criteria are identified for the various processbs* 
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Multiple Change Approach 
Bcyee, Jv. 



6.3 

p. 198 



Five kinds of vesulte^ six dimeneions, and four major aourcea of informal 
tion to be considered in evaluation are outlined. Results Include: Impersonal 
results, administrative results, results involving persons, changes in economic 
and social conditions, and changes in capital resources. The six dimensions 
include: kind of results sought, side effects produced, area of influence, 
time period within which the results are attained, cost of bringing about 
the results, and effect of factors other than the development project. The 
four types of data Include: already available data^ additional records, 
systematic observation, and systematic questioning. 



Multiple Dimensions of Program Effectiveness e.u 
Steele p. 200 

At least six dimensions of program effectiveness may need to be considered 
in evaluation— gr^wafvil reeutta, attcdttment of objectivess intent of the 
impaat and eignifiocmoe^ oontnbution to mission^ and use of reeouroee. 



National Assessment Program 5.1 



The national educational assessment activity provides a model for national 
evaluation. Data were collected on the extent to which American education was 
attaining its objectives. The objectives used were those negotiated from 
among those suggested by eahool ayateme, acholara of the diaoiplines and 
diaceming lay adulta. Emphasis was on the things that all individuala 
of a certain age should be able to do; what average youngateva of that 
age should be able to do, and things that the mat advanced can do. 
Sampling prooedurea were used both in selecting the behaviors that would be 
examined and in selecting the random group of youngsters to be tested. 



Natural Process Approach u.a 

S'jeele p, 1U3 

The purpose of evaluation is to help the programmer assess the value of 
programs and programming activity. Evaluation is an everyday process used by 
most human beings. The challenge is that of helping humans make a process 
that is subjective (within the mind) become more accurate. Program evaluation 
is a generic process in that the same basic activity is involved in evaluating 
the whole program or any part of it. This process has three elements: oHteria, 
evidence^ and judgmenta. Accuracy is increased by improving each of the three 
elementfi. 



p. 155 
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NEA Model 

Taba & Scojin 



5.3 

p. 179 



In an effort to broaden objectives based model, four major components 
are graphically presented as a square. Objectives are on the top of the 
square with evidence of pupil behavior pertaining to the objectives on the 
bottom. Evidence of teaching'-leaming operationa and evidence of noneahool 
faotOTB affecting learning complete the other two sides of the square. 



NewStart Evaluation System 1.5 
Lcomock^ Smith, A Warren p. 91 

Evaluation is essential in the development of packaged courses. The 
general scope of such evaluation is indicated by: topic areas where deoieiona 
are made, type of data base used in the deaieion, eource of data, and time 
of the decision. Formative/summative evaluation integrates different methods 
into a total research study. 



Program performance is evaluated by examining the effectiveness and 
efficiency of the program. Evaluation of both characteristics focus on 
the relationship of three key components: obsectiveei activitiee, and 
reeourcee* Symbolic patterns are developed by .using the first letter of each 
of the key components. Measures can be attained of program effectiveness, 
AO:PO; program efficiency, AO:AR; and activity efficiency, AO:AA. P stands 
for planned and A for actual. 



Ohio Model 5.3 
Starr et al. p. 177 

This model was designed on a statewide basis* It includes: program 
obdeotivee and meaeurdble goal etatemente; a data eet, inetrumentation, and 
procedures to measure the effectiveness of efforts to achieve specific program 
goals ; the means by which the evaluation system would be articulated with 
larger management systems of state divisions of vocational education. 
Attention was given to the time frame required for collecting data, the costs 
in carrying out the procedure, and the personnel required to manage the data. 



Model 

Deniston et al. 



5.3 

p. 173 



0 



Organizational Models 

Etizioni; Schulburg A Baker 



2.U 

p. 117 



Organizational models deal with an organization as a working unit capable 
of achieving goals. It includes both fimctione direct to the program and 
maintenanoe and support funations essential to the system. 
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Organization As a Total System 2.3 
ioung p. 109 

The total system needs to be understood to facilitate production. Or- 
ganization is a process with inputs and outputs; management represents the 
control or feedback mechanism. Design criteria are rules used to evaluate 
the ^acceptability of designs. Among the most widely used are roeasurability, 
feasibility, optiroality, reliability, and stability. 



Participant Reaction Approaches 1.7 

p. 98 

A variety of techniques including steering committees, analysis of 
attendance and re-^nrollment records, and end-of-*session reaction forms are 
used to get participant reactions to programs. Information is secured in 
terms of reactions to one or more of the following: oontent^ format^ 
facilities s teacher ^s performance ^ perception of values folloQ^up needed. 



Priority Decisions i.u 
Boyle p. 86 

Evaluation is important in setting program priorities. Criteria for 
making decisions about priorities are grouped into six broad categories: 
eociety'^comunityj clientele ^ political^ organizational^ resources j and 
personal. The specific criteria to be used must evolve from the key in- 
fluences affecting the particular program and programming situation. 



Program Contact System 2.u 

Duft p. 112 

This approach uses system analysis as a means of initiating a smooth, 
effective, and efficient flow of information from those providing it to 
those in need of it for decision purposes. The programming system includes 
four phases: program formulation^ decision phase j executions and evaluation 
of program results. The decision phase includes four subphases: program 
developments program strategy ^ program dissemination strategy ^ and program 
promotion strategy. 



Program Evaluation and Review Technique (PERT) 2.3 

p. 107 

Projects are broken into specific task components. Sequence and interface 
are identified through graphic presentation. Responsibility ^ time estimates^ 
and deadlines are established. Key steps and pivotal activities are 
identified. The flow chart provides a basis for analyzing the progress of 
work as the project progresses. 
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Program Planning Budgeting System (PPBS) 5.3 

p. 178 

PPBS has four major phases: (l) plcpming^ which is concerned with 
generating program objectives; (2) prograrmingj which is concerned with 
the preparation of al^ ^ative sets of activities and services to achieve 
program objectives; (o/ Jidgeting^ which is involved with the formulation 
of detailed plans, accounting, and reporting; and (f) evaluating j which 
is concerned with progress outputs and effectiveness of programs. 



Public Policy 6.7 
Berldk p. 205 

Program outcomes should be considered and classified as to whether they're 
just prograrmatio or whether they have pubtio policy ooneequenaee^ Public 
policy questions relate to moral valuBB. Evaluation that deals with public 
policy questions must be able to provide information related to the moral 
value questions that are involved. 



Research Models 

Weiee; Longeet; Cain A HollieteT 

The presence or absence of results 
research design. 



6.10 
p. 212 

can be detected by using a standard 



Social Indicators 6.9 
Pauleon ?• 210 

Agencies carrying out broad-based 9 many-faceted programs in a community 
might want to consider measuring results in terms of eo&ial indexee. The 
Community Health model, for example, uses measures on indexes in political, 
environmental, economic, social, and educational fields to analyze community 
situations as a basis for planning and then to use follow-up measures to examine 
results. Each of the major areas is examined by means of a groi^ of subscales* 



Social Systems Models 2. if 

Loomis 116 

Social systems models may have a good deal to contribute to program 
evaluation. One model includes three key kinds of ingredients: elemente^ 
proaeeeeBj and requiaitee for eocial aotione. Elements include: standards 9 . 
sanctions, knowledge, and attitudes which are key ingredients in evaluation 
Processes such as communication, institutionalization, boundary ma^intenance , 
systemic linkage, and social control affect both a program and its evaluation. 
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System Approach to Goal Setting 2.3 
Van Gigoh & Hill p. Ill 

System analysis is a process, a rational mode of approaching complex 
problems, a tool in decision making. A systems approach to goal setting 
includes: eyeteme definition^ etruotures ond deeign; ayetem deeign proaeee; 
defining the bonndariee of the eyetem and identifying how thoee within the 
ayetem judge ite performance; goal integration and methode of obtaining 
it; gOal-'aetting prooeaa* 



System Role Model 3.2 
Knox p. 12«* 

Seven conponents are identified from system and role theory and prior 
research and evaluation as the key targets of analysis in evaluation. They 
include: context , inputas proaeaa, atttivitys outcome judgmnts and application 
of findinga. The foilc^^ing points should be considered in designing evaluation 
systems: evidence, benefits, frequency, feedback^ ccMranitment, objectivity^ 
and standards. 



Trade-Off and Comparative Cost Approach 1.6 
Glaaa p. 95 

A format for evaluation of materials or educational activities includes: 
describing the product to be evaluated; evaluating the goale of the product; 
clari^ing the point of entry of the evaluator; determining the kinds of 
trade-^ffa involved; comparing coata with costs of altemativea; making an 
intrinaia^ (secondary) evaluation; making an outcome (primary) evaluation; 
forming yudgmenta cmd recormendationa; stipulating circumatancee that would 
modi^ the conclusions; and evaluating the evaluator* 



Transactional Evaluation 

Rippey 




Transactional evaluation looks at the effecta of changed programs on 
the changera themselves. It focuses on evaluating program acceptance and 
gives attention to examining: unanticipated as well as expected consequences, 
the effect on the total organization jind not just those directly involved, 
dislocation due to competition for resources. Both/ the eupportera and the 
critica of the program change are involved. '\ 
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Tylerian Models 5.1 

p. m 

Evaluation is concerned with determining whether education is actually 
producing the reeulte it eete out to achieve aa indicated by etatemente 
of objectivee* The evaluation process includes: analyzing objectives to 
identify and clarify content and behavior, identifying situations which 
will give the student a chance to express the behavior related to the 
content, selecting or developing instruments, analyzing the amount of change 
that has taken place in students^ analyzing the strengths and weaknesses 
of the program. 



Types of Evaluation 5 .a 

Wholey et at. P* 181 

An analysis of federal program evaluation came up with four categories 
of evaluation: program impact evaluations program strategy evaluation, 
project evaluation, and project rating* Monitoring, reporting systems, 
and cost analysis are defined but excluded from the categorization. Choice 
of type depends on importance and feasibility. 



Weighted Criteria Approach i«6 

Crane & Abt . p. 96 

Cost effectiveness of alternative curriculum materials is calculated by 
a detailed breakdown and analysis of componente, quality, and coat. The 
following elements are included: coverage, appropriateneee , motivational 
effeotiveneee, and coet. A score is computed using weights assigned to 
subcriteria. Scores of alternative products are compared. 



When-To-Do-It-Yourse1f Continuum 3*^ 
Alexander P^ 131 

Data-gathering activities are grouped into five main categories: habitual 
but unorganized, simple guides, reporting by etudente or teacher, poat^ or 
pre- and poet'-teeting, and experimental deeign using control or comparative 
group. Although program personnel are responsible for carrying out most 
of the activities, help should be gotten from specialists when the more 
complicated categories are involved^ 



Zones of Results ^ 6.2 

Lamrock, Smith, & Warren P* 195 

Results need to be examined Within the learning eetting, in real life 
during the course of the learning, and in real life at an internal after the 
completion of -tiie learning activity. 



Appendix B 

EVALUATION APPROACHES CATEGORIZED BY FIELD OF ORIGIN 



ADULT EDUCATION 



Adult Baeia Education 

Actual Component Approach [Knox, Hezirow, £ Darkenwald, Jr.] 
Improvement Evaluation [Kreitlow] 

Cormtunity Development 

Multiple Change Approach [Hayes, Jr.] 
Social Indicators [Paulson] 

Continuing Education 

System Role Model [Knox] 

Adoption of the Tyler Model by Adult Education 
Participant Reaction Approaches 

Extension Education 

Priority Decisions [Boyle] 

Developmental Evaluation — NewStart Evaluation System [Lamrock, Smith, £ Warren] 

Motivational Model [Lewis] 

Program Contact System [Duft] 

Management Information Systems (SEMIS) 

Means-Ends Hierarchy [Bennett] 

Natural Process Approach [Steele] 

Monitoring Evaluation [Bruce] 

Adoption of the Tyler Model by Adult Education 

Multiple Dimensions of Program Effectiveness [Steele] 

Effectiveness of Methods [Wilson 6 Gallup] 

When-To-Do-It-Yourself Continuum [Alexander] 
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Vocational Education 

Criteria and Objectives [Matteson] 

Locally Directed Evaluation [Byram £ Robertson] 

Ohio Model [Starr et al,] 

Information Domains [Nelson] 

Impact Evaluation [Borus £ Tash] 



OTHER FIELDS OF EDUCATION 

Higher Education 

Institutional Evaluation [Forehand] 

Materials Evaluation-^^Trade-Off and Comparative Cost Approach [Glass] 
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OTHER FIELDS OF EDUCATION (conij 

Elementary and Secondary Education 

Decision Centered Evaluation (CIPP) [Stufflebeam; Phi Delta Kappa] 
Discrepancy Evaluation [Provus] 

Developmental Evaluation-- IPI Formative Evaluation [Lindvall & Cox] 
Materials Evaluation—Weighted Criteria Approach [Crane fi Abt] 
Macro System Model [Alkin] 

System Approach to Goal Setting [Van Gigch 6 Hill] 

Countenance of Evaluation [Stake] 

Appraisal Model [Harris] 

Data Management tPhi Delta Kappa] 

Transactional Evaluation [Rippey] 

Tylerian Models and Adaptations [Tyler; Gottman £ Clasen; Menfin 6 Womer] 
Goal-Referenced Instruction [Popham £ Baker] 

Evaluation As Facilitation of Learning [Bloom, Hastings, £ Madaus] 

Objectives: Categorizations and Criteria 

Goal Ft»ee Evaluation [Scriven] 

Public Policy [Berlak] 

Accountability 



OTHER SOCIAL FIELDS 

Health 

Criteria of Success Approach [Suchman] 
0-A-R Model [Deniston et al,] 

Several Social Fields 

Differential Evaluation [Tripodi, Fellin, 6 Epstein] 
I3xecution-Impact Approach [Freeman £ Sherwood] 

Managerial Systems — ^Program Evaluation and Review Technique (PERT) 
Organization As a Total System [Young] 
Evaluating Decision Making [Hesseling] 
Socio-Organizational Systems 
Social Systems Models [Loomis] 

Organizational Models [Etizioni; Schulberg £ Baker] 
Types of Evaluation [Wholey et al>] 
Research Models [Weiss; Longest; Cain £ Robertson] 
Efficiency Examinations 
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Appendix C 

A TRY AT DEVELOPING A TAXONOMY OF PROGRAM EVALUATION APPROACHES 



The six groups used as the ordering device in this monograph were 
abstracted from this very rough and preliminary try tc classify the types 
of evaluation activities that go on in adult education. This is only one 
way in which approaches can be classified. Others need to be developed and 
comparative merits debated. The point presently isn't what organizational 
system to use, but the need for recognizing that the term evaluation includes 
such a number of different phenomena that classification systems are needed 
to help the practitioner and the theorist develop and improve evaluative 
activities. 

The very tentative classification system used here has several categories 

and subdivisions. Starting with the broadest categories first, in practice 

and in general discussion, there appear to t three general types of evaluation 

that take place in adult education; 

Evaluation of poptiaipante (grading, certification, etc.). 
Evaluation of pereonnel (merit raises, rank and promotion. Civil 

Service, performance reviews, etc.). 
Evaluation of programe. 

Although the three should interact closely, they often are treatt^d as 
quite distinctly different fields of specialized endeavor. 

Each domain has its own subcategories. Our concern currently is with 

the domain of progrcun evaluation. There are several categories within it. 

Some are well established and well recognized; others are emerging. Program 

evaluation presently appears to contain the following categories: 

Evaluation of prooeee. 
Evaluation of reeulte. 

Evaluation of program ae funotioning, produHng eyetme. 
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Eventually program evaluation may come to mean only the third category 
with the other two identified as other types of evaluation. 

Evaluation of process is excluded from this monograph. It would 
include such things as approaches to evaluating group interaction, student -teacher 
interaction, teacher performance, effectiveness of teaching strategy, quality 
of promotion, extensiveness of legitimation activities, effectiveness of use 
of visual aids, etc. A good deal has been done in establishing criteria and 
procedures for evaluating the processes and materials used in adult education. 

Evaluation of results has been included because of the general pre- 
occupation with this type of evaluation. It has been divided into two categories: 

Attainment of objeotivee. 
Evaluation of outaome and effeate. 

The first category includes the myriad of models that use objectives as 
the definition of what results will be judged against. The second category 
includes broader approaches to examining results. 

Subcategorizations within evaluation of program as funcl^ctiing, producing 
systems evolve from the content of the present models and approaches rather 
than from need or logic. Recent approaches to the program as a system appear 
to be of two kinds; 

Patterns of the program eyetm. 

Patterns of the program eyatem ae mirrowed in evaluation proaeeeee 
and data. 

Within patter-as of program systems, there seem to be at least two 

types of approaches emerging: 

Evaluation of the system and its components aa input into 

deaieion making. 
Evaluation of effectiveness of program parte. 



Patterns of program as mirrored in evaluative activities also can 

be further subdivided into at least two categories: 

Kinda of data; typee of aotivitiee. 
Evaluation proaeeeee. 

See Appendix Table C-1 for the composite outline of categories. As we 

better understand present approaches, develop new ones, and are better able 

to relate to program needs, it's possible that these categories will be 

replaced by others . 

Table C-1 

A SYSTEM FOR CLASSIFYING EVALUATION APPROACHES 

If Evaluation of participants. 
^ II. Evaluation of personnel. 
Illf Evaluation of program. 

A. Evaluation of: proc3SS. 

B. Evaluation of results. 

* 1. ReBulta — Attainment of Objectives. 

* 2. Reaulta-'Svalmtion of Outoomee and Effects. 

C. Evaluation of programs as functioning, producing systems. 
If Patterns of program systems. 

* af Evaluation Ae Input Into Decision Making. 

(1) Major or general decisions. 

(2) Specific decision situations (selection of materials, 
setting priorities, etc.). 

* b. Evaluation of Program Parte. 

2f Patterns of program systems as mirrored in evaluation. 

* a. Evaluation-^-^Kinda of Data} Types of Activities 

* bf Evaluation Processes. 

(1) General processes. 

(2) Specifically applied processes. 



* Indicates. the six categories that were used as major groupings for 
the organization of the monograph. Within level C, numerical levels were 
dropped to simplify the number of things the reader has to deal with. 
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It seems inadvisable to take space in this monograph to try to explain 
the various subcategories and differentiate between them. However, those 
familiar with some of the current models may find Appendix Table C-2 useful in 
exploring the categorization. It shows where I*d locate several of the emerging 
adult and extension education conceptualizations and some of the more well-known 
models from general education and health and welfare. See Table 2 on pages 44-^5 
for a complete listing of the approaches included in this monograph. 

You will note for example, that among the adult and extension education 
models, Knox and Bennett both are describing ways of categorizing evaluative 
data. How^ /er, they're describing different things in that Knox includes 
a wide range of components, while Bennett concentrates only on process and 
outcomes. Bruce, Steele, and Kreitlow all have posed framewoi'Ks dealing 
with evaluation processes. However, the three are dealing with such different 
things that all three could be used at the same time without undue duplication. 

And as you consider some of the better known conceptualizations in general 
education and related fields— those by Tyler, Suchman, Scriven, Stake, Stufflebeam— 
it*s readily apparent that they're talking about quite different things. You 
don^t ohooae among them to get a pcartioular thing dones hut ohooaea from 
them aooording to the different things that you need to do. Thus, you'd 
choose a Tyler model if you only wanted to examine the attainment of objectives; 
Suchman* s approach if you wanted to look at results in terms of objectives 
plus other criteria; Scriven if you wanted to examine more results than just 
those specified in the objective. You'd turn to Stake or Stufflebeam if you 
were interested in the system involved. Stufflebeam pDses the system in terms 
of types of evaluation which influence decision making. Stake describes system 
in terms of types of evaluative data that reflect the program components and 
the acts of evaluation. 



Table C-2 



LOCATION WITHIN THE CLASSIFICATION SYSTEM OF ADULT AND 
EXTENSION EDUCATION AND OTHER MAJOP APPROACHES 



III. Evaluation of program. 

B. Evaluation of results. 

1. Eeeults — Attainment of Objectives. 

a. Tylevian Models [Tyler 6 others]. 

b. Inatructiondl models including Goal'-Referenaed Instruction^ 
[Popham e Baker]. 

c. Program objectives approaches including Criteria and ObjeotiveB 
[Suchman]. 

2. Reeult8"Evaluation of Outcomes and Effects. 

a. Goal Free Evaluation [Scriven]. 

b. Research Models. 

c. Aocountdbility . 

d. Multiple Change Approach [Hayes, Jr.]. 

C. Evaluation of programs as functioning, producing systems. 

1. Patterne of program systems. 

a. Evaluation As Input Into Decision Making. 

(1) Major or general decisions. 

(a) Decision Centered Evaluation (CIPP) [Stufflebeam; 
Phi Delta Kappa]. 

(b) Differential Evaluation [Tripodi, Fellin, 6 Epstein]. 

(2) Specific decision situations. 

(a) Development Models such as IPI Formative Evaluation [Lindvall 6 
Cox], and Neu>Start Evaluation System [Lamrock, Smith 6 Warren]. 

(b) Discrepancy Evaluation [Provus]. 

b. Evaluation of Program Plane. 

(1) Actual Component Approach [Knox, Mezirow, S Darkenwald, Jr.]. 

(2) Execution-'Impact Approach [Freeman S Sherwood]. 

(3) Management models such as Program Evaluation and 
Review Technique (PERT). 

2. Patterns of evaluation. 

a. Evatuation-'-Kinds of Data; Types of Activities. 

(1) Countenance of Evaluation [Stake]. 

(2) System Rote Model [Knox]. 

(3) Means-Ends Hierarchy [Bennett]. 

b. Evaluation Processes. 

(1) General processes. 

(a) Appraisal Model [Harris]. 

(b) Natural Process Model [Steele]. 

(2) Specifically applied processes. 

(a) MonitoHng Evaluation [Bruce]. 

(b) Improvement Evaluation [Kreitlow]. 
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Categories can be crossed very easily with a bit of creativity. For 
example, although the program decision CIPP model was* designed as input into 
decision making, each of the four types could be considered components of 
program (context, input, process, and product) and used as a format for 
judging effectiveness. On the other hand, the actual component model was 
designed to examine effectiveness, but with a bit of restructuring it could 
be cast as a guide to key decisions. 

Classification systems can become extremely engrossing. They^re 
not valuable in and of themselves. They have value only if they help 
better understand the phenomena being classified. 
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